REVIEW OF VECTOR EMBEDDINGS FUSION METHODS

DOI: 10.31673/2412-4338.2022.048489

Abstract

This paper presents a comprehensive review of the state-of-the-art techniques for embeddings fusion in natural language processing and machine learning. Embeddings fusion refers to the task of combining multiple word or document embeddings into a single representation that can capture the different aspects of the input data. This is typically used in multimodal machine learning applications where inputs come from different sources with different formats or in situations when embeddings are already available and need to be combined in the model. The paper covers various fusion methods, including concatenation, averaging, weighted averaging, and neural network-based approaches.
A detailed analysis of the benefits and limitations of each method, as well as the scenarios in which they are most effective is provided. In the paper vector embeddings fusion methods are categorized by model architecture type as well as by fusion type. Moreover, recommendations to choose the optimal type of fusion method given task limitations are described. In addition, the paper discusses the evaluation metrics commonly used to assess the quality of fused embeddings, such as similarity and classification accuracy.
Overall, this review paper provides a valuable resource for researchers and practitioners in the field of natural language processing and machine learning who wish to deepen their understanding of embeddings fusion methods and their applications. The insights and recommendations presented in this paper can help guide the selection of appropriate fusion methods and improve the performance of various natural language processing and machine learning tasks. By staying up-to-date with the latest developments in embeddings fusion, researchers and practitioners can continue to push the boundaries of natural language processing and machine learning.

Keywords: machine learning, natural language processing, mathematical modelling, neural networks, word embeddings, document embeddings, embeddings fusion metrics.

References
1. Camacho-Collados J., Pilehvar M. Embeddings in Natural Language Processing. Proceedings of the 28th International Conference on Computational Linguistics: Tutorial Abstracts. 2020. P. 10 – 15. 2. Ramachandram D., Taylor G. W. Deep Multimodal Learning: A Survey on Recent Advances and Trends. IEEE Signal Processing Magazine. 2017. №34(6). P. 96 – 108.
3. Blandfort P., Karayil T., Raue F., Hees J., Dengel A.R. Fusion Strategies for Learning User Embeddings with Neural Networks. 2019 International Joint Conference on Neural Networks (IJCNN). 2019. P. 1 – 8.
4. Wróblewska A., Dąbrowski J., Pastuszak M., Michałowski A., Daniluk M., Rychalska B., Wieczorek M., Sysko-Romańczuk S. Designing Multi-Modal Embedding Fusion-Based Recommender. Electronics. 2022. №11(9). P. 1391. 5. Stahlschmidt S. R., Ulfenborg B., Synnergren J. Multimodal deep learning for biomedical data fusion: a review. Briefings in Bioinformatics. 2022. №23(2). P. 1 – 15. 6. Gao J., Li P., Chen Z., Zhang J. A survey on deep learning for multimodal data fusion. Neural Computation. 2020. №32(5). P. 829 – 864. 7. Shvetsova N., Chen B., Rouditchenko A., Thomas S., Kingsbury B., Feris R. S., Kuehne H. Everything at once-multi-modal fusion transformer for video retrieval. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022. P. 20020 – 20029. 8. Qiu S., Chang G. H., Panagia M., Gopal D. M., Au R., Kolachalama V. B. Fusion of deep learning models of MRI scans, Mini–Mental State Examination, and logical memory test enhances diagnosis of mild cognitive impairment. Alzheimer's & Dementia: Diagnosis, Assessment & Disease Monitoring. 2018. №10. P. 737 – 749. 9. McFee B., Lanckriet G., Jebara T. Learning Multi-modal Similarity. Journal of machine learning research. 2011. №12(2). P. 491 – 523
10. Li Q., Xiong Q., Gao M., Ji S., Yu Y., Wu C. Multi-view heterogeneous fusion and embedding for categorical attributes on mixed data. Soft Computing. 2020. №24. P. 10843 – 10863.