融合预训练和双级元蒸馏的医学短文本分类方法

doi:10.3778/j.issn.1002-8331.2310-0176

摘要/Abstract

摘要： 为解决医学文本使用预训练模型处理速度慢、对硬件的算力要求高、难以分辨少样本类别，以及使用传统小模型由于自身限制难以做到足够的准确度的问题，提出了一种融合预训练和元蒸馏的文本分类模型PTMD（fusion of pre-training and meta distillation model）。PTMD针对医学文本的多标签问题，通过对比训练对RoBERTa预训练方法进行微调，再由双向内置注意力简单循环单元充分获取语义信息。最后在传统蒸馏模型的基础上融合元学习和助教的思想，通过教学实验和双级模型等方法，提高模型的教学水平，最终在减少训练成本的基础上得到一个高性能医学文本分类模型。实验结果表明，教师模型在CHIP2019评测三数据集上的F1值达到了85.47%，同时学生模型在F1值损失1.45个百分点的情况下，将模型规模缩小到教师模型的近1/6，效果高于大多数传统预训练模型和知识蒸馏模型，证明了该模型具有良好的实用价值。

关键词: 文本分类, 预训练模型, 知识蒸馏

Abstract: In order to solve the problems of slow classification processing speed of pre-trained models, high computational power requirements for hardware, the difficulty in distinguishing small sample categories, and the inability of traditional small models to achieve sufficient accuracy due to their own limitations, a text classification model named PTMD (fusion of pre-training and meta distillation model) that integrates pre-training and meta distillation is proposed. PTMD focuses on the multi label problem of medical texts, fine-tunes the RoBERTA pre-training method through comparative training, and fully obtains semantic information through bidirectional built-in attention simple loop units. Finally, based on the traditional distillation model, the idea of meta learning is integrated. Through teaching experiments and other methods, the teaching level of the model is improved, knowledge transfer is completed, and a high-performance medical text classification model is obtained while reducing training costs.The experimental results show that the F1 value of the teacher model on the CHIP2019 evaluation dataset reaches 85.47%, while DistillBiLSTM reduces the model size to nearly 1/6 of the teacher model with a loss of 1.45?percentage points in the F1 value,and its performance is better than that of most traditional pre-training models and knowledge distillation models, demonstrating that this model has good practical value.

Key words: text classification, pre-trained models, knowledgedistillation

廖列法, 姜炫至. 融合预训练和双级元蒸馏的医学短文本分类方法[J]. 计算机工程与应用, 2025, 61(8): 117-125.

LIAO Liefa, JIANG Xuanzhi. Medical Short Text Classification Method Combining Pre-Training and Metadistillation[J]. Computer Engineering and Applications, 2025, 61(8): 117-125.

参考文献

[1] 周润民, 胡旭耀, 吴克伟, 等. 基于交叉注意力的方面级情感分析[J]. 计算机工程与应用, 2023, 59(9): 190-197.
ZHOU R M, HU X Y, WU K W, et al. Aspect-based sentiment analysis with cross-heads attention[J]. Computer Engineering and Applications, 2023, 59(9): 190-197.
[2] 赵萍, 窦全胜, 唐焕玲, 等. 融合词信息嵌入的注意力自适应命名实体识别[J]. 计算机工程与应用, 2023, 59(8): 167-174.
ZHAO P, DOU Q S, TANG H L, et al. Attention adaptive model with word information embeding for named entity recognition[J]. Computer Engineering and Applications, 2023, 59(8): 167-174.
[3] PETERS M, NEUMANN M, IYYER M, et al. Deep contextualized word representations[C]//Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2018: 2227-2237.
[4] BROWN T, MANN B, RYDER N, et al. Language models are few-shot learners[C]//Proceedings of the 34th International Conference on Neural Information Processing Systems, 2020: 1877-1901.
[5] SONG K, TAN X, QIN T, et al. MPNet: masked and permuted pre?training for language understanding[J]. arXiv:2004. 09297, 2020.
[6] LAN Z, CHEN M, GOODMAN S, et al. ALBERT: a lite BERT for self-supervised learning of language representations[C]//Proceedings of the International Conference on Learning Representations, 2020.
[7] 叶雪梅, 毛雪岷, 夏锦春, 等. 文本分类TF-IDF算法的改进研究[J]. 计算机工程与应用, 2019, 55(2): 104-109.
YE X M, MAO X M, XIA J C, et al. Improved approach to TF-IDF algorithm in text classification[J]. Computer Engineering and Applications, 2019, 55(2): 104-109.
[8] KIM Y. Convolutional neural networks for sentence classification[J]. arXiv:1408.5882, 2014.
[9] GUO X, ZHANG H, YANG H, et al. A single attention-based combination of CNN and RNN for relation classification[J]. IEEE Access, 2019, 7: 12467-12475.
[10] 田慧, 武杰, 边云, 等. 基于DenseNet结合迁移学习的胰腺囊性肿瘤分类方法[J]. 波谱学杂志, 2023, 40(3): 270-279.
TIAN H, WU J, BIAN Y, et al. Classification of pancreatic cystic tumors based on DenseNet and transfer learning[J]. Chinese Journal of Magnetic Resonance, 2023, 40(3): 270-279.
[11] 李洋, 董红斌. 基于CNN和BiLSTM网络特征融合的文本情感分析[J]. 计算机应用, 2018, 38(11): 3075-3080.
LI Y, DONG H B. Text sentiment analysis based on feature fusion of convolution neural network and bidirectional long short-term memory network[J]. Journal of Computer Applications, 2018, 38(11): 3075-3080.
[12] 白思萌, 牛振东, 何慧, 等. 基于超图注意力网络的生物医学文本分类方法[J]. 数据分析与知识发现, 2022, 6(11): 13-24.
BAI S M, NIU Z D, HE H, et al. Biomedical text classification method based on hypergraph attention network[J]. Data Analysis and Knowledge Discovery, 2022, 6(11): 13-24.
[13] DEVLIN J, CHANG M W, LEE K, et al. BERT: pre-training of deep bidirectional transformers for language understanding[J]. arXiv:1810.04805, 2018.
[14] VASWANI A, SHAZEER N, PARMAR N, et al. Attention is all you need[C]//Proceedings of the 31st International Conference on Neural Information Processing Systems, 2017: 6000-6010.
[15] 刘威, 马磊, 李凯, 等. 基于多粒度字形增强的中文医学命名实体识别[J]. 计算机工程, 2024, 50(2): 337-344.
LIU W, MA L, LI K, et al. Chinese medical named entity recognition based on multi-granularity glyph enhancement[J]. Computer Engineering, 2024, 50(2): 337-344.
[16] 李帅驰, 杨志豪, 王鑫雷, 等. 基于特征增强的开放域知识库问答系统[J]. 计算机工程与应用, 2022, 58(17): 206-212.
LI S C, YANG Z H, WANG X L, et al. Open domain Chinese knowledge based question answering based on feature enhancement[J]. Computer Engineering and Applications, 2022, 58(17): 206-212.
[17] 陆晓蕾, 倪斌. 基于预训练语言模型的BERT-CNN多层级专利分类研究[J]. 中文信息学报, 2021, 35(11): 70-79.
LU X L, NI B. BERT-CNN: a hierarchical patent classifier based on pre-trained language model[J]. Journal of Chinese Information Processing, 2021, 35(11): 70-79.
[18] LIU Y, OTT M, GOYAL N, et al. RoBERTa: a robustly optimized bert pretraining approach[J]. arXiv:1907.11692, 2019.
[19] 王廷轩, 刘韬, 王振亚, 等. 知识蒸馏与迁移学习的轴承故障诊断应用研究[J]. 计算机工程与应用, 2023, 59(13): 289-297.
WANG T X, LIU T, WANG Z Y, et al. Applied research on bearing fault diagnosis based on knowledge distillation and transfer learning[J]. Computer Engineering and Applications, 2023, 59(13): 289-297.
[20] 孙红, 黄瓯严. 基于知识蒸馏的短文本分类方法[J]. 软件导刊, 2021, 20(6): 23-27.
SUN H, HUANG O Y. Short text classification method based on knowledge distillation[J]. Software Guide, 2021, 20(6): 23-27.
[21] LEI T. When attention meets fast recurrence: training language models with reduced compute[J]. arXiv:2102.12459, 2021.
[22] ZHOU W, XU C, MCAULEY J. BERT learns to teach: knowledge distillation with meta learning[J]. arXiv:2106. 04570, 2021.
[23] XU Z. RoBERTa-wwm-ext fine-tuning for Chinese text classification[J]. arXiv:2103.00492, 2021.
[24] 宗辉, 张泽宇, 杨金璇, 等. 基于人工智能的中文临床试验筛选标准文本分类研究[J]. 生物医学工程学杂志, 2021, 38(1): 105-110.
ZONG H, ZHANG Z Y, YANG J X, et al. Artificial intelligence based Chinese clinical trials eligibility criteria classification[J]. Journal of Biomedical Engineering, 2021, 38(1): 105-110.
[25] 杨飞洪, 王序文, 李姣. 基于BERT-TextCNN模型的临床试验筛选短文本分类方法[J]. 中华医学图书情报杂志, 2021, 30(1): 54-59.
YANG F H, WANG X W, LI J. BERT-TextCNN-based classification of short texts from clinical trials[J]. Chinese Journal of Medical Library and Information Science, 2021, 30(1): 54-59.
[26] CUI Y, CHE W, LIU T, et al. Revisiting pre-trained models for Chinese natural language processing[C]//Findings of the Association for Computational Linguistics, 2020: 657-668.
[27] KINGMA D P, BA J. Adam: a method for stochastic optimization[J]. arXiv:1412.6980, 2014.
[28] 叶晨晨. 面向生物医学领域的引入外部信息的多标签文本分类研究[D]. 南京: 东南大学, 2022.
YE C C. Research on multi-label text classification with external information for biomedical donmain[D]. Nanjing: Southeast University, 2022.
[29] JOHNSON R, ZHANG T. Deep pyramid convolutional neural networks for text categorization[C]//Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, 2017: 562-570.