Abstract:

Magnetic Resonance Imaging（MRI） of the knee is the preferred method for diagnosing knee injuries. However, the interpretation of knee MRI is time-intensive and prone to errors. In order to assist clinicians to diagnose more accurately, a deep learning model based on multimodal fusion is proposed to detect general anomalies, Anterior Cruciate Ligament（ACL） tears and meniscus tears. HOG features and LBP features are contacted as low-level semantic features after selecting the features with over 95% contribution by PCA. Feature pyramid network on the basis of VGG16 model merges multiple feature maps as high-level semantic features. The low-level semantic features and the high-level semantic features are fused by correlation to get the prediction probability through an energy model which consists of multi-layer neural network. The experiments show that the AUC of the general anomaly, ACL tear and meniscus tear are 0.941?0, 0.970?8 and 0.8479, which have certain advantages over the results of the high-level semantic features or the low-level semantic features.

Key words: knee injury, Magnetic Resonance Imaging（MRI） image, multimodality, feature fusion

摘要：

膝关节磁共振成像（MRI）是诊断膝关节损伤的首选方法。然而，MRI影像的人工诊断是费时的，而且容易出现诊断错误。为了更准确地预测膝关节损伤，辅助临床医生做出诊断，提出一种多模态特征融合的深度学习模型，用于检测一般异常、前交叉韧带撕裂和半月板撕裂。提取梯度方向直方图（Histogram of Oriented Gradients， HOG）特征和局部二值模式（Local Binary Pattern，LBP）特征，经contact融合后利用PCA选取特征贡献度超过95%的特征作为传统特征；在VGG16模型的基础上加入金字塔融合的思想，将多个feature map的信息融合作为深度特征；将传统特征和深度特征经多层神经网络的能量模型进行相关性融合，作为多模态的特征，并得到预测概率。实验结果表明，上述模型在一般异常、前交叉韧带撕裂和半月板撕裂下ROC曲线下的面积（AUC）值分别为0.941?0、0.970?8和0.847?9，与传统特征和深度特征的效果相比，具有明显的优势，可以实现更有效的预测。

关键词: 膝关节损伤, 磁共振成像（MRI）影像, 多模态, 特征融合