融合多视图特征的放射学报告生成

doi:10.3778/j.issn.1002-8331.2402-0218

摘要/Abstract

摘要： 放射学报告生成涉及从多源图像中提取特征并将其转化为文本描述。当前的研究面临着多视图和不同长度报告的挑战，导致生成的临床报告准确性不足和语义不连贯。针对这些问题，提出了一种融合多视图特征的方法，通过从原始图像中进行多次局部特征提取和细粒度融合减少了信息丢失。通过标注工具获得并嵌入全局上下文表示，让模型在训练时使用更具概括性的文本，以获得更为流畅的描述。在IU X-Ray和MIMIC-CXR两个数据集上的实验表明，该方法在R2Gen模型上的应用使生成报告的质量得分平均提升了2.96个百分点。此外在自行构建的中文肺部CT报告数据集上进行了影像报告到诊断结论的生成实验，表现了该方法的通用性。

关键词: 放射学报告生成, 多视图, 细粒度融合, 全局上下文

Abstract: Radiology report generation involves extracting features from multiple source images and converting them into textual descriptions. The current research faces challenges related to multiple views and varying report lengths, resulting in insufficient accuracy and semantic incoherence in the generated clinical reports. To address these issues, a method that integrates features from multiple views is proposed, reducing information loss through multiple local feature extractions and fine-grained fusion from original images. Global context representation is obtained and embedded by using annotation tools, allowing the model to use more comprehensive text during training for smoother descriptions. Experiments on IU X-Ray and MIMIC-CXR datasets show an average improvement of 2.96 percentage points in report quality scores with the application of this method on the R2Gen model. Furthermore, experiments on a self-constructed Chinese lung CT report dataset for image report to diagnostic conclusion generation demonstrate the generality of the proposed method.

Key words: radiology report generation, multiple views, fine-grained fusion, global context

欧佳乐, 昝红英, 张坤丽, 师相龙, 马玉团. 融合多视图特征的放射学报告生成[J]. 计算机工程与应用, 2025, 61(10): 320-330.

OU Jiale, ZAN Hongying, ZHANG Kunli, SHI Xianglong, MA Yutuan. Radiology Report Generation Integrating Multi-View Features[J]. Computer Engineering and Applications, 2025, 61(10): 320-330.

参考文献

[1] DELRUE L, GOSSELIN R, ILSEN B, et al. Difficulties in the interpretation of chest radiography[M]//Comparative interpretation of CT and standard radiography of the chest. Berlin, Heidelberg: Springer, 2011: 27-49.
[2] GOERGEN S K, POOL F J, TURNER T J, et al. Evidence-based guideline for the written radiology report: methods, recommendations and implementation challenges[J]. Journal of Medical Imaging and Radiation Oncology, 2013, 57(1): 1-7.
[3] BRADY A, LAOIDE R ó, MCCARTHY P, et al. Discrepancy and error in radiology: concepts, causes and consequences[J]. The Ulster Medical Journal, 2012, 81(1): 3-9.
[4] VINYALS O, TOSHEV A, BENGIO S, et al. Show and tell: a neural image caption generator[C]//Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2015: 3156-3164.
[5] TRAN A, MATHEWS A, XIE L X. Transform and tell: entity-aware news image captioning[C]//Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2020: 13032-13042.
[6] JING B Y, XIE P T, XING E. On the automatic generation of medical imaging reports[C]//Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics. Stroudsburg: ACL, 2018: 2577-2586.
[7] LI C Y, LIANG X D, HU Z T, et al. Hybrid retrieval-generation reinforced agent for medical image report generation[C]//Proceedings of the 32nd International Conference on Neural Information Processing Systems, 2018: 1537-1547.
[8] LI C Y, LIANG X D, HU Z T, et al. Knowledge-driven encode, retrieve, paraphrase for medical image report generation[J]. Proceedings of the AAAI Conference on Artificial Intelligence, 2019, 33(1): 6666-6673.
[9] LOVELACE J, MORTAZAVI B. Learning to generate clinically coherent chest X-ray reports[C]//Findings of the Association for Computational Linguistics: EMNLP 2020. Stroudsburg: ACL, 2020: 1235-1243.
[10] CHEN Z H, SONG Y, CHANG T H, et al. Generating radiology reports via memory-driven transformer[C]//Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing. Stroudsburg: ACL, 2020: 1439-1449.
[11] MAO J H, XU W, YANG Y, et al. Deep captioning with multimodal recurrent neural networks (m-RNN)[J]. arXiv:1412.6632, 2014.
[12] TIAN J, LI C, SHI Z C, et al. A diagnostic report generator from CT volumes on liver tumor with semi-supervised attention mechanism[C]//Proceedings of the 21st International Conference on Medical Image Computing and Computer-Assisted Intervention. Cham: Springer, 2018: 702-710.
[13] LU J S, XIONG C M, PARIKH D, et al. Knowing when to look: adaptive attention via a visual sentinel for image captioning[C]//Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2017: 3242-3250.
[14] DONAHUE J, HENDRICKS L A, GUADARRAMA S, et al. Long-term recurrent convolutional networks for visual recognition and description[C]//Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2015: 2625-2634.
[15] ANDERSON P, HE X D, BUEHLER C, et al. Bottom-up and top-down attention for image captioning and visual question answering[C]//Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2018: 6077-6086.
[16] YANG X Y, YE M C, YOU Q Z, et al. Writing by memorizing: hierarchical retrieval-based medical report generation[C]//Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing. Stroudsburg: ACL, 2021: 5000-5009.
[17] CHEN Z H, SHEN Y L, SONG Y, et al. Cross-modal memory networks for radiology report generation[C]//Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing. Stroudsburg: ACL, 2021: 5904-5914.
[18] QIN H, SONG Y. Reinforced cross-modal alignment for radiology report generation[C]//Findings of the Association for Computational Linguistics: ACL 2022. Stroudsburg: ACL, 2022: 448-458.
[19] MIURA Y, ZHANG Y H, TSAI E, et al. Improving factual completeness and consistency of image-to-text radiology report generation[C]//Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Stroudsburg: ACL, 2021: 5288-5304.
[20] QI P, ZHANG Y H, ZHANG Y H, et al. Stanza: a Python natural language processing toolkit for many human languages[C]//Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations. Stroudsburg: ACL, 2020: 101-108.
[21] ZHANG T Y, KISHORE V, WU F, et al. BERTScore: evaluating text generation with BERT[J]. arXiv:1904.09675, 2019.
[22] LIU F L, GE S, WU X. Competence-based multimodal curriculum learning for medical report generation[C]//Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing. Stroudsburg: ACL, 2021: 3001-3012.
[23] LIU F L, YIN C C, WU X, et al. Contrastive attention for automatic chest X-ray report generation[C]//Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021. Stroudsburg: ACL, 2021: 269-280.
[24] ZHANG Y X, WANG X S, XU Z Y, et al. When radiology report generation meets knowledge graph[J]. Proceedings of the AAAI Conference on Artificial Intelligence, 2020, 34(7): 12910-12917.
[25] DEMNER-FUSHMAN D, KOHLI M D, ROSENMAN M B, et al. Preparing a collection of radiology examinations for distribution and retrieval[J]. Journal of the American Medical Informatics Association, 2016, 23(2): 304-310.
[26] JOHNSON A E W, POLLARD T J, BERKOWITZ S J, et al. MIMIC-CXR, a de-identified publicly available database of chest radiographs with free-text reports[J]. Scientific Data, 2019, 6(1): 317.
[27] SIMONYAN K, ZISSERMAN A. Very deep convolutional networks for large-scale image recognition[J]. arXiv:1409. 1556, 2014.
[28] HE K M, ZHANG X Y, REN S Q, et al. Deep residual learning for image recognition[C]//Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2016: 770-778.
[29] HUANG G, LIU Z, VAN DER MAATEN L, et al. Densely connected convolutional networks[C]//Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2017: 2261-2269.
[30] SINGH D, KUMAR V, KAUR M. Densely connected convolutional networks-based COVID-19 screening model[J]. Applied Intelligence, 2021, 51(5): 3044-3051.
[31] KE Q H, BENNAMOUN M, AN S J, et al. A new representation of skeleton sequences for 3D action recognition[C]//Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2017: 4570-4579.
[32] YANG H Y, HUANG S J, DAI X Y, et al. Fine-grained knowledge fusion for sequence labeling domain adaptation[C]//Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing. Stroudsburg: ACL, 2019: 4195-4204.
[33] VASWANI A, SHAZEER N, PARMAR N, et al. Attention is all you need[C]//Advances in Neural Information Processing Systems 30, 2017: 5998-6008.
[34] NOORALAHZADEH F, GONZALEZ N P, FRAUENFELDER T, et al. Progressive transformer-based generation of radiology reports[C]//Findings of the Association for Computational Linguistics: EMNLP 2021. Stroudsburg: ACL, 2021: 2824-2832.
[35] 何玉洁, 杜方, 史英杰, 等. 基于深度学习的命名实体识别研究综述[J]. 计算机工程与应用, 2021, 57(11): 21-36.
HE Y J, DU F, SHI Y J, et al. Survey of named entity recognition based on deep learning[J]. Computer Engineering and Applications, 2021, 57(11): 21-36.
[36] 常洪阳, 昝红英, 马玉团, 等. 脑卒中疾病电子病历实体及实体关系标注语料库构建[J]. 中文信息学报, 2022, 36(8): 37-45.
CHANG H Y, ZAN H Y, MA Y T, et al. Corpus construction for named-entity and entity relations for electronic medical records of stroke disease[J]. Journal of Chinese Information Processing, 2022, 36(8): 37-45.
[37] CHANG H Y, ZAN H Y, ZHANG S, et al. Corpus construction for named-entity and entity relations for electronic medical records of cardiovascular disease[C]//Proceedings of the 8th China Conference on Health Information Processing. Singapore: Springer, 2022: 3-18.
[38] IRVIN J, RAJPURKAR P, KO M, et al. CheXpert: a large chest radiograph dataset with uncertainty labels and expert comparison[J]. Proceedings of the AAAI Conference on Artificial Intelligence, 2019, 33(1): 590-597.
[39] PAPINENI K, ROUKOS S, WARD T, et al. BLEU: a method for automatic evaluation of machine translation[C]//Proceedings of the 40th Annual Meeting on Association for Computational Linguistics. Stroudsburg: ACL, 2002: 311-318.
[40] DENKOWSKI M, LAVIE A, DENKOWSKI M, et al. Meteor 1.3: automatic metric for reliable optimization and evaluation of machine translation systems[C]//Proceedings of the 6th Workshop on Statistical Machine Translation, 2011: 85-91.
[41] LIN C Y. ROUGE: a package for automatic evaluation of summaries[M]//Text summarization branches out. Stroudsburg: ACL, 2004.
[42] VEDANTAM R, ZITNICK C L, PARIKH D. CIDEr: consensus-based image description evaluation[C]//Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2015: 4566-4575.
[43] RENNIE S J, MARCHERET E, MROUEH Y, et al. Self-critical sequence training for image captioning[C]//Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2017: 1179-1195.
[44] WANG Z Y, LIU L Q, WANG L, et al. METransformer: radiology report generation by transformer with multiple learnable expert tokens[C]//Proceedings of the 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2023: 11558-11567.
[45] JING B Y, WANG Z Y, XING E. Show, describe and conclude: on exploiting the structure information of chest X-ray reports[C]//Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. Stroudsburg: ACL, 2019: 6570-6580.