Abstract:

With the rapid development of information technology, information exists in various forms and sources. Different forms of existence or information sources can be referred to as one modal, and data composed of two or more modalities is called multi-modal data. Multi-modal data fusion is responsible for effectively integrating the information of multiple modalities, absorbing the advantages of different modalities, and completing the integration of information. Natural phenomena have very rich characteristics, and it is difficult for a single mode to provide complete information about a certain phenomenon. Faced with the fusion requirements of maintaining the diversity and completeness of the modal information after fusion, maximizing the advantages of each modal, and reducing the information loss caused by the fusion process, how to integrate the information of each modal has become a new challenge that exists in many fields. This paper briefly describes common multimodal fusion methods and fusion architectures, summarizes three common fusion models, and briefly analyzes the advantages and disadvantages of the three architectures of collaboration, joint, and codec, as well as specific fusion methods such as multi-core learning and image models. In the application of multi-modality, it analyzes and summarizes multi-modal video clip retrieval, comprehensive multi-modal information generation content summary, multi-modal sentiment analysis, and multi-modal man-machine dialogue system. The paper also proposes the current problems of multi-modal fusion and the future research directions.

Key words: multimodal, multimodal fusion, multimodal fusion architecture, machine learning, neural network

摘要：

随着当今信息技术的飞速发展，信息的存在形式多种多样，来源也十分广泛。不同的存在形式或信息来源均可被称之为一种模态，由两种或两种以上模态组成的数据称之为多模态数据。多模态数据融合负责将多个模态的信息进行有效的整合，汲取不同模态的优点，完成对信息的整合。自然现象具有十分丰富的特征，单一模态很难提供某个现象的完整信息。面对保持融合后具有各个模态信息的多样性以及完整性、使各个模态的优点最大化、减少融合过程造成的信息损失等方面的融合要求，如何对各个模态的信息进行融合成为了多个领域广泛存在的一个新挑战。简要阐述了常见的多模态融合方法、融合架构，总结了三个常见的融合模型，简要分析协同、联合、编解码器三大架构的优缺点以及多核学习、图像模型等具体融合方法。在多模态的应用方面，对多模态视频片段检索、综合多模态信息生成内容摘要、多模态情感分析、多模态人机对话系统进行了分析与总结。指出了当前多模态融合出现的问题，并提出未来的研究方向。

关键词: 多模态, 多模态融合, 多模态融合架构, 机器学习, 神经网络