Review of Multi-Modal Retrieval in Medicine

doi:10.3778/j.issn.1002-8331.2208-0102

Abstract

Abstract: With the rapid application of computing and big data technologies in medicine and the gradual improvement of medical information storage standards, data in medicine are growing explosively. The medical data are stored in a multi-modal form, and these multi-modal data are often stored simultaneously and in a complementary way, so it is of great clinical significance to implement the retrieval of multi-modal data in medicine. In this paper, the current status about multi-modal retrieval in medicine is reviewed firstly, and the approaches for multi-modal retrieval in medicine can be categorized as text-based, content-based and fused information-based. Content-based multi-modal retrieval is further classified into traditional feature-based and deep feature-based. Then, the commonly used performance evaluation indexes such as precision, recall and average precision mean are introduced. Finally, this paper analyzes the challenges of multi-modal retrieval in medicine, and provides a prospect for the future multi-modal retrieval in medicine.

Key words: multi-modal retrieval, retrieval approach, multi-modal datasets in medicine, performance evaluation

摘要： 随着计算机与大数据技术在医学领域中的迅速应用以及医疗信息存储标准的逐渐完善，医学数据呈爆炸式增长。医学数据由于其自身特点而呈现出多模态形式，且这些多模态数据往往同时出现、互相补充，因此实现多模态数据间的相互检索具有重要的临床价值。回顾了近年来多模态检索在医学领域的实现方法，将其归纳为基于文本、基于内容以及基于融合信息的多模态检索，基于内容的多模态检索可进一步划分为基于传统特征的检索和基于深度特征的检索。针对多模态检索算法的性能，介绍了准确率、召回率以及平均精度均值等常用的评价指标。分析了当前医学领域多模态检索所面临的挑战，并对未来医学领域多模态检索的研究发展进行了展望。

关键词: 多模态检索, 检索方法, 医学多模态数据集, 性能评价

DING Guohui, ZHANG Qi, FANG Shichao, LI Qing, SUN Xiaoyu, ZHANG Luxia, KONG Guilan. Review of Multi-Modal Retrieval in Medicine[J]. Computer Engineering and Applications, 2023, 59(1): 26-36.

丁国辉, 张琦, 房士超, 李青, 孙小宇, 张路霞, 孔桂兰. 多模态检索在医学领域的研究综述[J]. 计算机工程与应用, 2023, 59(1): 26-36.

References

[1] 杜鹏飞，李小勇，高雅丽.多模态视觉语言表征学习研究综述[J].软件学报，2021，32（2）：327-348.
DU P F，LI X Y，GAO Y L.Survey on multimodal visual language representation learning[J].Journal of Software，2021，32（2）：327-348.
[2] MüLLER H，UNAY D.Retrieval from and understanding of large-scale multi-modal medical datasets：A review[J].IEEE Transactions on Multimedia，2017，19（9）：2093-2104.
[3] 吴飞，朱文武，于俊清.多媒体技术研究：2014——深度学习与媒体计算[J].中国图象图形学报，2015，20（11）：1423-1433.
WU F，ZHU W W，YU J Q.Researches on multimedia technology 2014—deep learning and multimedia computing[J].Journal of Image and Graphics，2015，20（11）：1423-1433.
[4] 姚伟娜.基于深度哈希算法的图像—文本跨模态检索研究[D].北京：北京交通大学，2018.
YAO W N.Research on image-text cross-modal retrieval based on deep hash algorithm[D].Beijing：Beijing Jiaotong University，2018.
[5] ANAYI Y，KOGAN I，GELBART E，et al.A comparative study for chest radiograph image retrieval using binary texture and deep learning classification[C]//Proceedings of the 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society，2015：2940-2943.
[6] SMEULDERS A W M，WORRING M，SANTINI S，et al.Content-based image retrieval at the end of the early years[J].IEEE Transactions on Pattern Analysis and Machine Intelligence，2000，22（12）：1349-1380.
[7] CLOUGH P，SANDERSON M.The CLEF 2003 cross language image retrieval track[C]//Proceedings of the Workshop of the Cross-Language Evaluation Forum for European Languages，2003：581-593.
[8] CLOUGH P，SANDERSON M，MULLER H.The CLEF cross language image retrieval track（ImageCLEF） 2004[C]//Proceedings of the International Conference on Image and Video Retrieval，2004：243-251.
[9] WEI C H，LI Y，LI C T.Effective extraction of Gabor features for adaptive mammogram retrieval[C]//Proceedings of the IEEE International Conference on Multimedia and Expo，2007：1503-1506.
[10] GREENSPAN H，PINHAS A T.Medical image categorization and retrieval for PACS using the GMM-KL framework[J].IEEE Transactions on Information Technology in Biomedicine，2007，11（2）：190-202.
[11] DEMNER-FUSHMAN D.Annotation and retrieval of clinically relevant images[J].international Journal of Medical Informatics，2009，78（12）：59-67.
[12] AYED A B，KARDOUCHI M，SELOUNI S A.Rotation invariant fuzzy shape contexts based on eigenshapes and fourier transforms for efficient radiological image retrieval[C]//Proceedings of the International Conference on Multimedia Computing and Systems，2012：266-271.
[13] CAMLICA Z，TIZHOOSH H R，KHALVATI F.Medical image classification via SVM using LBP features from saliency-based folded data[C]//Proceedings of the IEEE 14th International Conference on Machine Learning and Applications（ICMLA），2015：128-132.
[14] CAMLICA Z，TIZHOOSH H R，KHALVATI F.Autoencoding the retrieval relevance of medical images[C]//Proceedings of the International Conference on Image Processing Theory，Tools and Applications（IPTA），2015：550-555.
[15] HEARST M A，DIVOLI A，GUTURU H，et al.BioText search engine：Beyond abstract search[J].Bioinformatics，2007，23（16）：2196-2197.
[16] KAHN C E.Multilingual retrieval of radiology images[J].Radiographics，2009，29（1）：23-29.
[17] XU S H，MCCUSKER J，KRAUTHAMMER M.Yale Image Finder（YIF）：A new search engine for retrieving biomedical images[J].Bioinformatics，2008，24（17）：1968-1970.
[18] STATHOPOULOS S，LOURENTZOU I，KVRIAKOPOULOU A，et al.IPL at CLEF 2013 medical retrieval task[C]//Proceedings of CLEF（Working Notes），2013.
[19] KALPATHY-CRAMER J.Effectiveness of global features for automatic medical image classification and retrieval—the experiences of OHSU at ImageCLEF med[J].Pattern Recognition Letters，2008，29（15）：2032-2038.
[20] ZHOU X，STERN R，MULLER H.Case-based fracture image retrieval[J].International Journal of Computer Assisted Radiology and Surgery，2012，7（3）：401-411.
[21] LOWE D G.Distinctive image features from scale-invariant keypoints[J].International Journal of Computer Vision，2004，60（2）：91-110.
[22] LIU S，LIU S，PUJOL S，et al.Propagation graph fusion for multi-modal medical content-based retrieval[C]//Proceedings of the 13th International Conference on Control Automation Robotics & Vision，2014：849-854.
[23] KUMAR A，KIM J，FENG D，et al.Graph-based retrieval of multi-modality medical images：A comparison of representations using simulated images[C]//Proceedings of the 25th IEEE International Symposium on Computer-Based Medical Systems，2012：1-6.
[24] KITANOVSKI I，STREZOSKI G，DIMITROVSKI I，et al.Multimodal medical image retrieval system[J].Multimedia Tools and Applications，2017，76（2）：2955-2978.
[25] DENG J，DONG W，SOCHER R，et al.ImageNet：A large-scale hierarchical image database[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2009：248-255.
[26] KALCHBRENNER N，BLUNSOM P.Recurrent continuous translation models[C]//Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing，2013：1700-1709.
[27] SUTSKEVER I，VINVALS O，LE Q V.Sequence to sequence learning with neural networks[C]//Advances in Neural Information Processing Systems，2014.
[28] GEHRING J，AULI M，GRANGIER D，et al.Convolutional sequence to sequence learning[C]//Proceedings of the International Conference on Machine Learning，2017：1243-1252.
[29] VINVALS O，TOSHEY A，BENGIO S，et al.Show and tell：A neural image caption generator[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2015：3156-3164.
[30] YAN F，MIKOLAJCZYK K.Deep correlation for matching images and text[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2015：3441-3450.
[31] FENG F X，WANG X J，LI R F，et al.Correspondence autoencoders for cross-modal retrieval[J].ACM Transactions on Multimedia Computing，Communications，and Applications，2015，12（S1）：1-22.
[32] ANDREW G，ARORA R，BILMES J A，et al.Deep canonical correlation analysis[C]//Proceedings of the 30th International Conference on International Conference on Machine Learning，2013：1247-1255.
[33] WANG W，YANG X Y，OOI B C，et al.Effective deep learning-based multi-modal retrieval[J].The VLDB Journal，2016，25（1）：79-101.
[34] JANOWCZYL A，MADABHUSHI A.Deep learning for digital pathology image analysis：A comprehensive tutorial with selected use cases[J].Journal of Pathology Informatics，2016，7（1）：29.
[35] SHI Y，CHEN S，YOU X，et al.Deep supervised information bottleneck Hashing for cross-modal retrieval based computer-aided diagnosis[J].arXiv：2205.08365，2022.
[36] MBILINYI A，SCHULDT H.Cross-modality medical image retrieval with deep features[C]//Proceedings of the IEEE International Conference on Bioinformatics and Biomedicine（BIBM），2020：2632-2639.
[37] ROSSI A，HOSSEINZADEH M，BIANCHINI M，et al.Multi-modal Siamese network for diagnostically similar lesion retrieval in prostate MRI[J].IEEE Transactions on Medical Imaging，2021，40（3）：986-995.
[38] CHUNG Y A，WENG W H.Learning deep representations of medical images using siamese CNNs with application to content-based image retrieval[J].arXiv：1711.08490，2017.
[39] ZAGORUYKO S，KOMODAKIS N.Learning to compare image patches via convolutional neural networks[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2015：4353-4361.
[40] CHOPRA S，HADSELL R，LECUN Y.Learning a similarity metric discriminatively，with application to face verification[C]//Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition，2005：539-546.
[41] HADSELL R，CHPPRA S，LECUN Y.Dimensionality reduction by learning an invariant mapping[C]//Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition，2006：1735-1742.
[42] KUMAR S，UDUPA R.Learning Hash functions for cross-view similarity search[C]//Proceedings of the 22nd International Joint Conference on Artificial Intelligence，2011.
[43] DING G，GUO Y，ZHOU J.Collective matrix factorization hashing for multimodal data[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2014：2075-2082.
[44] LIN Z J，DING G G，HAN J G，et al.Cross-view retrieval via probability-based semantics-preserving hashing[J].IEEE Transactions on Cybernetics，2017，47（12）：4342-4355.
[45] ZHOU J L，DING G G，GUO Y C.Latent semantic sparse hashing for cross-modal similarity search[C]//Proceedings of the 37th International ACM SIGIR Conference on Research & Development in Information Retrieval，2014：415-424.
[46] ZHANG D，LI W J.Large-scale supervised multimodal hashing with semantic correlation maximization[C]//Proceedings of the AAAI Conference on Artificial Intelligence，2014.
[47] 欧卫华，刘彬，周永辉，等.跨模态检索研究综述[J].贵州师范大学学报（自然科学版），2018，36（2）：114-120.
OU W H，LIU B，ZHOU Y H，et al.A survey on the cross-modal retrieval research[J].Journal of Guizhou Normal University（Natural Sciences），2018，36（2）：114-120.
[48] WANG W，OOI B C，YANG X，et al.Effective multi-modal retrieval based on stacked auto-encoders[J].Proceedings of the VLDB Endowment，2014，7（8）：649-660.
[49] CAO Y，LONG M，WANG J，et al.Correlation autoencoder hashing for supervised cross-modal search[C]//Proceedings of the 2016 ACM on International Conference on Multimedia Retrieval，2016：197-204.
[50] JIANG Q Y，LI W J.Deep cross-modal hashing[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2017：3232-3240.
[51] YANG E K，LIU M X，YAO D R，et al.Deep Bayesian hashing with center prior for multi-modal neuroimage retrieval[J].IEEE Transactions on Medical Imaging，2021，40（2）：503-513.
[52] ZHANG Y，OU W H，SHI Y F，et al.Deep medical cross-modal attention hashing[J].World Wide Web，2022，25（4）：1519-1536.
[53] 赵晓乐.面向胸部CT图像—文本的跨模态哈希检索技术研究[D].昆明：昆明理工大学，2019.
ZHAO X L.Research on cross-modal hash retrieval technology for chest CT image-text[D].Kunming：Kunming University of Science and Technology，2019.
[54] YU X，YU S，PRINCIPE J C.Deep deterministic information bottleneck with matrix-based entropy functional[C]//Proceedings of the IEEE International Conference on Acoustics，Speech and Signal Processing，2021：3160-3164.
[55] XU L M，ZENG X H，ZHENG B C，et al.Multi-manifold deep discriminative cross-modal hashing for medical image retrieval[J].IEEE Transactions on Image Processing：A Publication of the IEEE Signal Processing Society，2022，31：3371-3385.
[56] MARTíN-VALDIVIA M T.Using information gain to improve multi-modal information retrieval systems[J].Information Processing & Management，2008，44（3）：1146-1158.
[57] VIKRAM M，ANANTHARAMAN A，BS S，et al.An approach for multimodal medical image retrieval using latent dirichlet allocation[C]//Proceedings of the ACM India Joint International Conference on Data Science and Management of Data，2019：44-51.
[58] CAO Y，STEFFEY S，HE J B，et al.Medical image retrieval：a multimodal approach[J].Cancer Informatics，2015，13：125-136.
[59] 杨柳青，王冲.基于多特征融合的异质信息搜索推荐算法研究[J].计算机工程与应用，2022，58（13）：171-176.
YANG L Q，WANG C.Research on heterogeneous information search recommendation algorithm based on multifeature fusion[J].Computer Engineering and Applications，2022，58（13）：171-176.
[60] JOHNSON A，POLLARD T，MARK R，et al.Mimic-CXR database[J].PhysioNet10，2019，13026：C2JT1Q.
[61] JOHNSON A E W，POLLARD T J，BERKOWITZ S J，et al.MIMIC-CXR，a de?identified publicly available database of chest radiographs with free-text reports[J].Scientific Data，2019，6：1-8.
[62] JOHNSON A，LUNGREN M，PENG Y，et al.MIMIC-CXR-JPG-chest radiographs with structured labels[J/OL].PhysioNet（2019）[2022-07-01].https：//doi.org/10.13026/8360-t248.
[63] WANG X，PENG Y，LU L，et al.Chestx-ray8：Hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2017：2097-2106.
[64] 孙钦佩.肺结节CT图像病理特性智能分析与基于图像特征的信息检索关键技术研究[D].北京：中国科学院大学（中国科学院上海技术物理研究所），2017.
SUN Q P.Intelligent analysis of pathological characteristics of pulmonary nodules CT images and key techniques of information retrieval based on image feature[D].Beijing：University of Chinese Academy of Sciences（Shanghai Institute of Technical Physics，Chinese Academy of Sciences），2017.
[65] ELGER B S.Strategies for health data exchange for secondary，cross-institutional clinical research[J].Computer Methods and Programs in Biomedicine，2010，99（3）：230-251.
[66] HUGHES J，HUNTER D，SHEEHAN M，et al.European textbook on ethics in research[M].[S.l.]：Publications Office of European Union，2010.
[67] UNAV D，EKIN A，HU F.Medical image search and retrieval for improved tele?healthcare[M]//Telehealthcare computing and engineering.Boca Raton，FL，USA：CRC Press，2013：660-679.