LLMs与ML优势互补：政务回复质量检测及可解释的算法框架

doi:10.3778/j.issn.1002-8331.2412-0021

摘要/Abstract

摘要： 为了克服传统政务平台回复的人工质量检测难以满足庞大市民政务需求的问题，以及以文本特征为主的机器学习方法缺乏可解释性的问题，研究采用了一种结合大语言模型（large language models，LLMs）和机器学习（machine learning，ML）模型，并以非文本特征为主的政务回复质量检测算法框架。实验表明该算法能达到比其他单一使用LLMs或ML的方法更高的准确度，接近政务问答质量的人工判断结果，并且收敛速度快，训练成本更低。同时采用SHAP（Shapley additive explanations）框架提高了该方法的可解释性，结合八种非文本特征的重要性和对最终结果的贡献度，识别出影响政务问答质量的关键因素和特征作用机制，以此筛选得到高质量回复和提高平台服务质量，增进市民对政务问答平台的满意度和信任度。

关键词: 电子政务, 回复质量检测, 大语言模型, 机器学习, 特征工程

Abstract: To overcome the issues of the manual quality detection of digital government platform responses not being able to meet the vast demands of citizens for government services and the lack of interpretability in text-feature-based machine learning methods, this study adopts a government response quality detection algorithm framework that combines large language models (LLMs) with machine learning (ML) models, mainly based on non-text features. Experiments show that this method can achieve higher accuracy than other methods solely using LLMs or ML, approaching the results of manual evaluation on the quality of government responses, with faster convergence and lower training costs. The study also introduces the SHAP (Shapley additive explanations) model to enhance the interpretability of the proposed method, integrating the importance ranking of eight non-text features and their impact on the quality detection results, identifying the significant features that can improve the quality of government responses and feature mechanisms, which helps to efficiently select high-quality replies and improve the service quality of government platforms, promoting citizens?? trust in digital government platforms.

Key words: digital government, response quality detection, large language models, machine learning, feature engineering

方岢愿, 许珂维. LLMs与ML优势互补：政务回复质量检测及可解释的算法框架[J]. 计算机工程与应用, 2025, 61(16): 146-159.

FANG Keyuan, XU Kewei. Complementary Strengths of LLMs and ML: Government Service Response Quality Detection and Explanation Algorithm Framework[J]. Computer Engineering and Applications, 2025, 61(16): 146-159.

参考文献

[1] KLIEVINK B, JANSSEN M. Coordinating e-government service delivery[C]//Proceedings of the 11th Annual International Digital Government Research Conference on Public Administration Online: Challenges and Opportunities. New York: ACM, 2010: 209-216.
[2] NICULESCU A I, KUKANOV I, WADHWA B. DigiMo-towards developing an emotional intelligent chatbot in Singapore[C]//Proceedings of the 2020 Symposium on Emerging Research from Asia and on Asian Contexts and Cultures. New York: ACM, 2020: 29-32.
[3] 张洋. 电子政务平台网络安全问题与管理技术研究[J]. 网络安全和信息化, 2023(12): 132-134.
ZHANG Y. Research on network security and management technology of e-government platform[J]. Cybersecurity & Informatization, 2023(12): 132-134.
[4] ALI AL-SAI Z, ABUALIGAH L M. Big data and E-government: a review[C]//Proceedings of the 2017 8th International Conference on Information Technology. Piscataway: IEEE, 2017: 580-587.
[5] PAPADOMICHELAKI X, MENTZAS G. E-GovQual: a multiple-item scale for assessing e-government service quality[J]. Government Information Quarterly, 2012, 29(1): 98-109.
[6] ALRUWAIE M, EL-HADDADEH R, WEERAKKODY V. Citizens’ continuous use of EGovernment services: the role of self-efficacy, outcome expectations and satisfaction[J]. Government Information Quarterly, 2020, 37(3): 101485.
[7] CORBETT E, LE DANTEC C A. Going the distance: trust work for citizen participation[C]//Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. New York: ACM, 2018: 1-13.
[8] RODRíGUEZ P G, BURGUETE J L V, VAUGHAN R, et al. Quality dimensions in the public sector: municipal services and citizen??s perception[J]. International Review on Public and Nonprofit Marketing, 2009, 6(1): 75-90.
[9] SCOTT J K. Assessing the quality of municipal government web sites[J]. State and Local Government Review, 2005, 37(2): 151-165.
[10] JIANG X, JI S B. E-government web portal adoption: the effects of service quality[J]. e-Service Journal, 2014, 9(3): 43-60.
[11] STEPHEN J. The non-public encounter: self-service and the ephemoralisation of public service[M]. London: Palgrave Macmillan, 2021: 125-150.
[12] LV Z H, LI X M, WANG W X, et al. Government affairs service platform for smart city[J]. Future Generation Computer Systems, 2018, 81: 443-451.
[13] WIMMER M A. Integrated service modelling for online one-stop government[J]. Electronic Markets, 2002, 12(3): 149-156.
[14] FANG K Y, XU K W. Automating government response to citizens’ questions: a large language model-based question-answering guidance generation system[C]//Proceedings of the 2023 3rd International Conference on Digital Society and Intelligent Systems. Piscataway: IEEE, 2023: 386-389.
[15] 陈小昆, 左航旭, 廖彬, 等. 融合XGBoost与SHAP的冠心病预测及其特征分析模型[J]. 计算机应用研究, 2022, 39(6): 1796-1804.
CHEN X K, ZUO H X, LIAO B, et al. Coronary artery disease prediction and feature analysis model based on XGBoost and SHAP[J]. Application Research of Computers, 2022, 39(6): 1796-1804.
[16] LOPARDO G, GARREAU D. Comparing feature importance and rule extraction for interpretability on text data [C]//Proceedings of the 2023 International Workshops and Challenges on Pattern Recognition, Computer Vision, and Image Processing. Cham: Springer, 2023: 261-268.
[17] SEJNOWSKI T J. Large language models and the reverse turing test[J]. Neural Computation, 2023, 35(3): 309-342.
[18] YANG L D, HUANG B H, GUO S Q, et al. A small-sample text classification model based on pseudo-label fusion clustering algorithm[J]. Applied Sciences, 2023, 13(8): 4716.
[19] QUAZI S, MUSA S M. Performing text classification and categorization through unsupervised learning[C]//Proceedings of the 2023 1st International Conference on Advanced Engineering and Technologies. Piscataway: IEEE, 2023: 1-6.
[20] LIU X, WANG S, LU S Y, et al. Adapting feature selection algorithms for the classification of Chinese texts[J]. Systems, 2023, 11(9): 483.
[21] KOOCHEMESHKIAN P, IHOU KOFFI E, BOUGUILA N. Hidden variable models in text classification and sentiment analysis[J]. Electronics, 2024, 13(10): 1859.
[22] ALI S I M, NIHAD M, SHARAF H M, et al. Machine learning for text document classification-efficient classification approach[J]. IAES International Journal of Artificial Intelligence, 2024, 13(1): 703.
[23] ABASI A K, KHADER A T, AL-BETAR M A, et al. An improved text feature selection for clustering using binary grey wolf optimizer[C]//Proceedings of the 11th National Technical Seminar on Unmanned System Technology. Singapore: Springer, 2020: 503-516.
[24] ABUALIGAH L, ALSALIBI B, SHEHAB M, et al. A parallel hybrid krill herd algorithm for feature selection[J]. International Journal of Machine Learning and Cybernetics, 2021, 12(3): 783-806.
[25] THIRUMOORTHY K, MUNEESWARAN K. Optimal feature subset selection using hybrid binary Jaya optimization algorithm for text classification[J]. Sādhanā, 2020, 45(1): 201.
[26] ZHAO X Y, AN Y X, XU N, et al. Variational continuous label distribution learning for multi-label text classification[J]. IEEE Transactions on Knowledge and Data Engineering, 2024, 36(6): 2716-2729.
[27] STOICA P, BABU P. Pearson-Matthews correlation coefficients for binary and multinary classification[J]. Signal Processing, 2024, 222: 109511.
[28] DEVARAJA G, VARDHNI K, DHARSHITA R, et al. A comparative and analytical study of text classification models using various metrics and visualizations[C]//Proceedings of the 2023 OITS International Conference on Information Technology. Piscataway: IEEE, 2023: 1-6.
[29] LI S J, YUAN G H, YANG M, et al. Improving semi-supervised text classification with dual meta-learning[J]. ACM Transactions on Information Systems, 2024, 42(4): 1-28.
[30] THANGARASU G, ALLA K R. Detection of cyberbullying tweets in Twitter media using random forest classification[C]//Proceedings of the 2023 IEEE 13th Symposium on Computer Applications & Industrial Electronics. Piscataway: IEEE, 2023: 113-117.
[31] BEHERA S, KOLIPAKULA N, SWETHA P, et al. Text classification using machine learning algorithms on encrypted data[C]//Proceedings of the 2023 IEEE International Conference on Electronics, Computing and Communication Technologies. Piscataway: IEEE, 2023: 1-6.
[32] HARTMANN J, HUPPERTZ J, SCHAMP C, et al. Comparing automated text classification methods[J]. International Journal of Research in Marketing, 2019, 36(1): 20-38.
[33] ZEBERGA K, ATTIQUE M, SHAH B, et al. A novel text mining approach for mental health prediction using Bi-LSTM and BERT model[J]. Computational Intelligence and Neuroscience, 2022: 7893775.
[34] ZHU J, CAO Y Q. Research on performance enhancement strategies for multimodal short text classification by fusing BERT and CNNs[C]//Proceedings of the 2024 9th International Symposium on Computer and Information Processing Technology. Piscataway: IEEE, 2024: 464-469.
[35] YU Y, LIU X Z. Research on enterprise text classification methods of BiLSTM and CNN based on BERT[C]//Proceedings of the 2023 9th International Conference on Computing and Artificial Intelligence. New York: ACM, 2023: 491-495.
[36] THANANUKHUN K, JAIYEN S, JITKAJORNWANICH K, et al. Question classification for Thai conversational chatbots using artificial neural networks and multilingual BERT models[C]//Proceedings of the 2023 15th International Conference on Knowledge and Smart Technology. Piscataway: IEEE, 2023: 1-4.
[37] LI M G, LI W R, WANG F, et al. Applying BERT to analyze investor sentiment in stock market[J]. Neural Computing and Applications, 2021, 33(10): 4663-4676.
[38] JEON J, CROFT W B, LEE J H, et al. A framework to predict the quality of answers with non-textual features[C]//Proceedings of the 29th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. New York: ACM, 2006: 228-235.
[39] STRONG D M, LEE Y W, WANG R Y. Data quality in context[J]. Communications of the ACM, 1997, 40(5): 103-110.
[40] ZHU X L, GAUCH S. Incorporating quality metrics in centralized/distributed information retrieval on the World Wide Web[C]//Proceedings of the 23rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. New York: ACM, 2000: 288-295.
[41] MESA-VIEIRA C, GONZALEZ-JARAMILLO N, DíAZ-RíOS C, et al. Urban governance, multisectoral action, and civic engagement for population health, wellbeing, and equity in urban settings: a systematic review[J]. International Journal of Public Health, 2023, 68: 1605772.
[42] LIU Y, ARARAL E, WU J N. Policy responsiveness and its administrative organisation in China[J]. Policy & Politics, 2024, 52(3): 360-383.
[43] YADAV B. Citizen participation in government decision-making[J]. Research Review International Journal of Multidisciplinary, 2024, 9(4): 267-271.
[44] BEWINDA R, RAHMADANY A F, ACHMAD M. Community complaint services through electronic complaint channels in increasing the effectiveness of government affairs compulsory basic services[J]. Journal of Humanities and Social Sciences Studies, 2024, 6(9): 89-97.
[45] MENG T G, YANG Z S. Variety of responsive institutions and quality of responsiveness in cyber China[J]. China Review, 2020, 20(3): 13-42.
[46] WOOD D, HARMS P D, LOWMAN G H, et al. Response speed and response consistency as mutually validating indicators of data quality in online samples[J]. Social Psychological and Personality Science, 2017, 8(4): 454-464.
[47] LUNDBERG S, LEE S. A unified approach to interpreting model predictions[C]//Proceedings of the 31st International Conference on Neural Information Processing Systems, 2017: 4768-4777.
[48] CHOI E, BORKOWSKI M, ZAKOIAN J, et al. Utilizing content moderators to investigate critical factors for assessing the quality of answers on brainly, social learning Q&A platform for students: a pilot study[J]. Proceedings of the Association for Information Science and Technology, 2015, 52(1): 1-4.