计算机工程与应用 ›› 2023, Vol. 59 ›› Issue (1): 26-36.DOI: 10.3778/j.issn.1002-8331.2208-0102

• 热点与综述 • 上一篇    下一篇

多模态检索在医学领域的研究综述

丁国辉,张琦,房士超,李青,孙小宇,张路霞,孔桂兰   

  1. 1.北京大学 健康医疗大数据国家研究院,北京 100191
    2.沈阳航空航天大学 计算机学院,沈阳 110136
    3.浙江省北大信息技术高等研究院,杭州 311215
  • 出版日期:2023-01-01 发布日期:2023-01-01

Review of Multi-Modal Retrieval in Medicine

DING Guohui, ZHANG Qi, FANG Shichao, LI Qing, SUN Xiaoyu, ZHANG Luxia, KONG Guilan   

  1. 1.National Institute of Health Data Science, Peking University, Beijing 100191, China
    2.School of Computer Science, Shenyang Aerospace University, Shenyang 110136, China
    3.Advanced Institute of Information Technology, Peking University, Hangzhou 311215, China
  • Online:2023-01-01 Published:2023-01-01

摘要: 随着计算机与大数据技术在医学领域中的迅速应用以及医疗信息存储标准的逐渐完善,医学数据呈爆炸式增长。医学数据由于其自身特点而呈现出多模态形式,且这些多模态数据往往同时出现、互相补充,因此实现多模态数据间的相互检索具有重要的临床价值。回顾了近年来多模态检索在医学领域的实现方法,将其归纳为基于文本、基于内容以及基于融合信息的多模态检索,基于内容的多模态检索可进一步划分为基于传统特征的检索和基于深度特征的检索。针对多模态检索算法的性能,介绍了准确率、召回率以及平均精度均值等常用的评价指标。分析了当前医学领域多模态检索所面临的挑战,并对未来医学领域多模态检索的研究发展进行了展望。

关键词: 多模态检索, 检索方法, 医学多模态数据集, 性能评价

Abstract: With the rapid application of computing and big data technologies in medicine and the gradual improvement of medical information storage standards, data in medicine are growing explosively. The medical data are stored in a multi-modal form, and these multi-modal data are often stored simultaneously and in a complementary way, so it is of great clinical significance to implement the retrieval of multi-modal data in medicine. In this paper, the current status about multi-modal retrieval in medicine is reviewed firstly, and the approaches for multi-modal retrieval in medicine can be categorized as text-based, content-based and fused information-based. Content-based multi-modal retrieval is further classified into traditional feature-based and deep feature-based. Then, the commonly used performance evaluation indexes such as precision, recall and average precision mean are introduced. Finally, this paper analyzes the challenges of multi-modal retrieval in medicine, and provides a prospect for the future multi-modal retrieval in medicine.

Key words: multi-modal retrieval, retrieval approach, multi-modal datasets in medicine, performance evaluation