多模态驾驶员情绪识别研究综述

doi:10.3778/j.issn.1002-8331.2410-0153

摘要/Abstract

摘要： 准确识别驾驶员情绪可以有效预防潜在的危险驾驶行为，减少交通事故的发生，是提升道路安全和驾驶体验的重要技术。随着人工智能和多模态数据处理技术的进步，情绪识别技术从单模态方法逐步发展为多模态方法。梳理了当前多模态驾驶员情绪识别的研究进展，重点总结了面部表情、语音信号、生理信号以及车辆行为四种模态的识别流程，关键步骤包括数据预处理、特征提取和多模态融合。通过分析现有研究，总结了不同方法的优势与不足，介绍了多个驾驶员情绪相关数据集。最后结合当前研究所面临的挑战，提出了未来多模态驾驶员情绪识别研究领域的五个研究方向。

关键词: 驾驶员情绪识别, 多模态融合, 特征提取, 情感计算

Abstract: Accurately identifying driver emotions can effectively prevent potential dangerous driving behaviors and reduce the occurrence of traffic accidents. It is an important technology to improve road safety and driving experience. With the progress of artificial intelligence and multi-modal data processing technology, emotion recognition technology has gradually developed from a single-modal approach to a multi-modal approach. This paper reviews the current research progress of multi-modal driver emotion recognition, and focuses on the recognition process of facial expression, voice signal, physiological signal and vehicle behavior. The key steps include data preprocessing, feature extraction and multi-modal fusion. By analyzing the existing research, the advantages and disadvantages of different methods are summarized, and several driver emotion-related datasets are introduced. Finally, combined with the current research challenges, five research directions in the field of multi-modal driver emotion recognition in the future are proposed.

Key words: driver emotion recognition, multi-modal fusion, feature extraction, emotional computing

周欣颖, 李雷孝, 林浩, 张虎成. 多模态驾驶员情绪识别研究综述[J]. 计算机工程与应用, 2025, 61(10): 1-18.

ZHOU Xinying, LI Leixiao, LIN Hao, ZHANG Hucheng. Review of Multi-Modal Driver Emotion Recognition[J]. Computer Engineering and Applications, 2025, 61(10): 1-18.

参考文献

[1] SHAMS Z, NADERI H, NASSIRI H. Assessing the effect of inattention-related error and anger in driving on road accidents among Iranian heavy vehicle drivers[J]. IATSS Research, 2021, 45(2): 210-217.
[2] GALDERISI S. The need for a consensual definition of mental health[J]. World Psychiatry, 2024, 23(1): 52-53.
[3] ZAMAN K, SUN Z Y, SHAH S M, et al. Driver emotions recognition based on improved faster R-CNN and neural architectural search network[J]. Symmetry, 2022, 14(4): 687.
[4] LI W B, XUE J Y, TAN R C, et al. Global-local-feature-fused driver speech emotion detection for intelligent cockpit in automated driving[J]. IEEE Transactions on Intelligent Vehicles, 2023, 8(4): 2684-2697.
[5] KUO J, KOPPEL S, CHARLTON J L, et al. Evaluation of a video-based measure of driver heart rate[J]. Journal of Safety Research, 2015, 54: 55-59.
[6] ALKINANI M H, KHAN W Z, ARSHAD Q. Detecting human driver inattentive and aggressive driving behavior using deep learning: recent advances, requirements and open challenges[J]. IEEE Access, 2020, 8: 105008-105030.
[7] SAADI I, CUNNINGHAM D W, TALEB-AHMED A, et al. Driver’s facial expression recognition: a comprehensive survey[J]. Expert Systems with Applications, 2024, 242: 122784.
[8] ZEPF S, HERNANDEZ J, SCHMITT A, et al. Driver emotion recognition for intelligent vehicles: a survey[J]. ACM Computing Surveys, 2020, 53(3): 1-30.
[9] SOLEYMANI M, GARCIA D, JOU B, et al. A survey of multimodal sentiment analysis[J]. Image and Vision Computing, 2017, 65: 3-14.
[10] EKMAN P. Expression and the nature of emotion[M]//Approaches to emotion. New York: Psychology Press, 1984: 319-343.
[11] KLINEBERG O. Human emotions by Carroll E. Izard (review)[J]. Leonardo, 1981, 14(1): 69.
[12] WUNDT W. Vorlesungen über die menschen-und thierseele[M]. Berlin: Springer, 1990: 103.
[13] RUSSELL J A, MEHRABIAN A. Evidence for a three-factor theory of emotions[J]. Journal of Research in Personality, 1977, 11(3): 273-294.
[14] HINTON G E, OSINDERO S, TEH Y W. A fast learning algorithm for deep belief nets[J]. Neural Computation, 2006, 18(7): 1527-1554.
[15] NGIAM J, KHOSLA A, KIM M, et al. Multimodal deep learning[C]//Proceedings of the 28th International Conference on Machine Learning, 2011: 689-696.
[16] HUANG Y, DU C Z, XUE Z H, et al. What makes multi-modal learning better than single (provably)[C]//Proceedings of the 35th Conference on Neural Information Processing Systems, 2021: 10944-10956.
[17] SRIVASTAVA N, SALAKHUTDINOV R. Multimodal learning with deep Boltzmann machines[J]. Journal of Machine Learning Research, 2014, 15(1): 2949-2980.
[18] TZIRAKIS P, TRIGEORGIS G, NICOLAOU M A, et al. End-to-end multimodal emotion recognition using deep neural networks[J]. IEEE Journal of Selected Topics in Signal Processing, 2017, 11(8): 1301-1309.
[19] BALTRUSAITIS T, AHUJA C, MORENCY L P. Multimodal machine learning: a survey and taxonomy[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2019, 41(2): 423-443.
[20] VIOLA P, JONES M J. Robust real-time face detection[J]. International Journal of Computer Vision, 2004, 57: 137-154.
[21] GUO G J, WANG H Z, YAN Y, et al. A fast face detection method via convolutional neural network[J]. Neurocomputing, 2020, 395: 128-137.
[22] LI X, LAI S Q, QIAN X M. DBCFace: towards pure convolutional neural network face detection[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2022, 32(4): 1792-1804.
[23] ZHANG Z P, LUO P, LOY C C, et al. Facial landmark detection by deep multi-task learning[C]//Proceedings of the 13th European Conference on Computer Vision. Cham: Springer, 2014: 94-108.
[24] YUE X Q, LI J, WU J, et al. Multi-task adversarial autoencoder network for face alignment in the wild[J]. Neurocomputing, 2021, 437: 261-273.
[25] EBRAHIMI KAHOU S, MICHALSKI V, KONDA K, et al. Recurrent neural networks for emotion recognition in video[C]//Proceedings of the 2015 ACM International Conference on Multimodal Interaction. New York: ACM, 2015: 467-474.
[26] PITALOKA D A, WULANDARI A, BASARUDDIN T, et al. Enhancing CNN with preprocessing stage in automatic emotion recognition[J]. Procedia Computer Science, 2017, 116: 523-529.
[27] KUO C M, LAI S H, SARKIS M. A compact deep learning model for robust facial expression recognition[C]//Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. Piscataway: IEEE, 2018: 2202-2208.
[28] LU Y S, LIU C S, CHANG F L, et al. JHPFA-net: joint head pose and facial action network for driver yawning detection across arbitrary poses in videos[J]. IEEE Transactions on Intelligent Transportation Systems, 2023, 24(11): 11850-11863.
[29] DENG J K, GUO J, VERVERAS E, et al. RetinaFace: single-shot multi-level face localisation in the wild[C]//Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2020: 5202-5211.
[30] DOUKAS M C, VERVERAS E, SHARMANSKA V, et al. Free-HeadGAN: neural talking head synthesis with explicit gaze control[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023, 45(8): 9743-9756.
[31] ZHAO X C, WANG L Z, SUN J X, et al. HAvatar: high-fidelity head avatar via facial model conditioned neural radiance field[J]. ACM Transactions on Graphics, 2023, 43(1): 1-16.
[32] KOELSTRA S, MUHL C, SOLEYMANI M, et al. DEAP: a database for emotion analysis using physiological signals[J]. IEEE Transactions on Affective Computing, 2012, 3(1): 18-31.
[33] HOUSSEIN E H, HAMMAD A, ALI A A. Human emotion recognition from EEG-based brain—computer interface using machine learning: a comprehensive review[J]. Neural Computing and Applications, 2022, 34(15): 12527-12557.
[34] GUPTA V, CHOPDA M D, PACHORI R B. Cross-subject emotion recognition using flexible analytic wavelet transform from EEG signals[J]. IEEE Sensors Journal, 2019, 19(6): 2266-2274.
[35] ALICKOVIC E, KEVRIC J, SUBASI A. Performance evaluation of empirical mode decomposition, discrete wavelet transform, and wavelet packed decomposition for automated epileptic seizure detection and prediction[J]. Biomedical Signal Processing and Control, 2018, 39: 94-102.
[36] PRUIM R H R, MENNES M, BUITELAAR J K, et al. Evaluation of ICA-AROMA and alternative strategies for motion artifact removal in resting state fMRI[J]. NeuroImage, 2015, 112: 278-287.
[37] JIRAYUCHAROENSAK S, PAN-NGUM S, ISRASENA P. EEG-based emotion recognition using deep learning network with principal component based covariate shift adaptation[J]. The Scientific World Journal, 2014: 627892.
[38] TURAKHIA M P, DESAI M, HEDLIN H, et al. Rationale and design of a large-scale, app-based study to identify cardiac arrhythmias using a smartwatch: the Apple Heart Study[J]. American Heart Journal, 2019, 207: 66-75.
[39] RAY D, COLLINS T, WOOLLEY S I, et al. A review of wearable multi-wavelength photoplethysmography[J]. IEEE Reviews in Biomedical Engineering, 2021, 16: 136-151.
[40] ELGENDI M, FLETCHER R, LIANG Y B, et al. The use of photoplethysmography for assessing hypertension[J]. NPJ Digital Medicine, 2019, 2: 60.
[41] WANG W J, DEN BRINKER A C, STUIJK S, et al. Algorithmic principles of remote PPG[J]. IEEE Transactions on Bio-Medical Engineering, 2017, 64(7): 1479-1491.
[42] 高绮煌, 谢凯, 贺正方, 等. 复杂环境下多模态特征融合的疲劳驾驶检测[J]. 电子测量技术, 2023, 46(6): 106-115.
GAO Q H, XIE K, HE Z F, et al. Fatigue driving detection with multi-modal feature fusion in complex environments[J]. Electronic Measurement Technology, 2023, 46(6): 106-115.
[43] CHENG J, WANG P, SONG R C, et al. Remote heart rate measurement from near-infrared videos based on joint blind source separation with delay-coordinate transformation[J]. IEEE Transactions on Instrumentation and Measurement, 2020, 70: 5005313.
[44] AK?AY M B, O?UZ K. Speech emotion recognition: emotional models, databases, features, preprocessing methods, supporting modalities, and classifiers[J]. Speech Communication, 2020, 116: 56-76.
[45] 乔栋, 陈章进, 邓良, 等. 基于改进语音处理的卷积神经网络中文语音情感识别方法[J]. 计算机工程, 2022, 48(2): 281-290.
QIAO D, CHEN Z J, DENG L, et al. Method for Chinese speech emotion recognition based on improved speech processing convolutional neural network[J]. Computer Engineering, 2022, 48(2): 281-290.
[46] CHEN C X, ZHANG P Y. TRNet: two-level refinement network leveraging speech enhancement for noise robust speech emotion recognition[J]. Applied Acoustics, 2024, 225: 110169.
[47] NGUYEN T H, LU D N, NGUYEN D N, et al. Dynamic basic activity sequence matching method in abnormal driving pattern detection using smartphone sensors[J]. Electronics, 2020, 9(2): 217.
[48] CHEN H L, ZHAO X H, LI Z L, et al. Construction and analysis of driver takeover behavior modes based on situation awareness theory[J]. IEEE Transactions on Intelligent Vehicles, 2024, 9(2): 4040-4054.
[49] SHI X P, WONG Y D, LI M Z, et al. A feature learning approach based on XGBoost for driving assessment and risk prediction[J]. Accident Analysis & Prevention, 2019, 129: 170-179.
[50] ALEESA R S, MAHVASH MOHAMMADI H, MONADJEMI A, et al. Dataset classification: an efficient feature extraction approach for grammatical facial expression recognition[J]. Computers and Electrical Engineering, 2023, 110: 108891.
[51] PAN B, HIROTA K, JIA Z Y, et al. Multimodal emotion recognition based on feature selection and extreme learning machine in video clips[J]. Journal of Ambient Intelligence and Humanized Computing, 2023, 14(3): 1903-1917.
[52] DU G L, WANG Z Y, GAO B Y, et al. A convolution bidirectional long short-term memory neural network for driver emotion recognition[J]. IEEE Transactions on Intelligent Transportation Systems, 2021, 22(7): 4570-4578.
[53] MORALES-VARGAS E, REYES-GARCíA C A, PEREGRINA-BARRETO H. On the use of action units and fuzzy explanatory models for facial expression recognition[J]. PLoS One, 2019, 14(10): e0223563.
[54] PATIL M, VENI S. Driver emotion recognition for enhancement of human machine interface in vehicles[C]//Proceedings of the 2019 International Conference on Communication and Signal Processing. Piscataway: IEEE, 2019: 420-424.
[55] BAKHEET S, AL-HAMADI A. A framework for instantaneous driver drowsiness detection based on improved HOG features and na?ve Bayesian classification[J]. Brain Sciences, 2021, 11(2): 240.
[56] ZHANG Y, ZHANG L, HOSSAIN M A. Adaptive 3D facial action intensity estimation and emotion recognition[J]. Expert Systems with Applications, 2015, 42(3): 1446-1464.
[57] 权学良, 曾志刚, 蒋建华, 等. 基于生理信号的情感计算研究综述[J]. 自动化学报, 2021, 47(8): 1769-1784.
QUAN X L, ZENG Z G, JIANG J H, et al. Physiological signals based affective computing: a systematic review[J]. Acta Automatica Sinica, 2021, 47(8): 1769-1784.
[58] JENKE R, PEER A, BUSS M. Feature extraction and selection for emotion recognition from EEG[J]. IEEE Transactions on Affective Computing, 2014, 5(3): 327-339.
[59] 王跃飞, 马伟丽, 王文康, 等. 基于生理特征映射的驾驶员情绪在线识别模型构建方法[J]. 机械工程学报, 2022, 58(20): 379-390.
WANG Y F, MA W L, WANG W K, et al. Online recognition model construction method of driver emotion based on physiological feature mapping[J]. Journal of Mechanical Engineering, 2022, 58(20): 379-390.
[60] HALIM Z, REHAN M. On identification of driving-induced stress using electroencephalogram signals: a framework based on wearable safety-critical scheme and machine learning[J]. Information Fusion, 2020, 53: 66-79.
[61] NI J, XIE W Y, LIU Y P, et al. Driver emotion recognition involving multimodal signals: electrophysiological response, nasal-tip temperature, and vehicle behavior[J]. Journal of Transportation Engineering, Part A: Systems, 2024, 150(1): 04023125.
[62] SAHAYADHAS A, SUNDARAJ K, MURUGAPPAN M, et al. Physiological signal based detection of driver hypovigilance using higher order spectra[J]. Expert Systems with Applications, 2015, 42(22): 8669-8677.
[63] PANAHI F, RASHIDI S, SHEIKHANI A. Application of fractional Fourier transform in feature extraction from electrocardiogram and galvanic skin response for emotion recognition[J]. Biomedical Signal Processing and Control, 2021, 69: 102863.
[64] MATEOS-GARCíA N, GIL-GONZáLEZ A-B, LUIS-REBOREDO A, et al. Driver stress detection from physiological signals by virtual reality simulator[J]. Electronics, 2023, 12(10): 2179.
[65] DU N, YANG X J, ZHOU F. Psychophysiological responses to takeover requests in conditionally automated driving[J]. Accident Analysis & Prevention, 2020, 148: 105804.
[66] 刘聪, 万根顺, 高建清, 等. 基于韵律特征辅助的端到端语音识别方法[J]. 计算机应用, 2023, 43(2): 380-384.
LIU C, WAN G S, GAO J Q, et al. End-to-end speech recognition method based on prosodic features[J]. Journal of Computer Applications, 2023, 43(2): 380-384.
[67] BEN ALEX S, MARY L, BABU B P. Attention and feature selection for automatic speech emotion recognition using utterance and syllable-level prosodic features[J]. Circuits, Systems, and Signal Processing, 2020, 39(11): 5681-5709.
[68] ANIKIN A. A moan of pleasure should be breathy: the effect of voice quality on the meaning of human nonverbal vocalizations[J]. Phonetica, 2020, 77(5): 327-349.
[69] RAJISHA T M, SUNIJA A P, RIYAS K S. Performance analysis of Malayalam language speech emotion recognition system using ANN/SVM[J]. Procedia Technology, 2016, 24: 1097-1104.
[70] MISHRA S P, WARULE P, DEB S. Speech emotion recognition using MFCC-based entropy feature[J]. Signal, Image and Video Processing, 2024, 18(1): 153-161.
[71] JIA X Y, SHEN X X. Multimodal emotion distribution learning[J]. Cognitive Computation, 2022, 14(6): 2141-2152.
[72] 沈燕, 肖仲喆, 李冰洁, 等. 采用GW-MFCC模型空间参数的语音情感识别[J]. 计算机工程与应用, 2015, 51(10): 219-222.
SHEN Y, XIAO Z Z, LI B J, et al. Speech emotion recognition using GW-MFCC feature[J]. Computer Engineering and Applications, 2015, 51(10): 219-222.
[73] 李牧, 杨宇恒, 柯熙政. 基于混合特征提取与跨模态特征预测融合的情感识别模型[J]. 计算机应用, 2024, 44(1): 86-93.
LI M, YANG Y H, KE X Z. Emotion recognition model based on hybrid-Mel Gama frequency cross-attention transformer modal[J]. Journal of Computer Applications, 2024, 44(1): 86-93.
[74] PAVLIDIS I, DCOSTA M, TAAMNEH S, et al. Dissecting driver behaviors under cognitive, emotional, sensorimotor, and mixed stressors[J]. Scientific Reports, 2016, 6: 25651.
[75] SHI Y, BOFFI M, PIGA B E A, et al. Perception of driving simulations: can the level of detail of virtual scenarios affect the driver’s behavior and emotions?[J]. IEEE Transactions on Vehicular Technology, 2022, 71(4): 3429-3442.
[76] WANG K X, AN N, LI B N, et al. Speech emotion recognition using Fourier parameters[J]. IEEE Transactions on Affective Computing, 2015, 6(1): 69-75.
[77] KOLLIAS D, ZAFEIRIOU S. Exploiting multi-CNN features in CNN-RNN based dimensional emotion recognition on the OMG in-the-wild dataset[J]. IEEE Transactions on Affective Computing, 2021, 12(3): 595-606.
[78] ZAMAN K, SUN Z Y, SHAH B, et al. A novel driver emotion recognition system based on deep ensemble classification[J]. Complex & Intelligent Systems, 2023, 9(6): 6927-6952.
[79] RASTGOO M N, NAKISA B, MAIRE F, et al. Automatic driver stress level classification using multimodal deep learning[J]. Expert Systems with Applications, 2019, 138: 112793.
[80] NAM Y, LEE C. Cascaded convolutional neural network architecture for speech emotion recognition in noisy conditions[J]. Sensors, 2021, 21(13): 4399.
[81] MOU L T, ZHAO Y Y, ZHOU C, et al. Driver emotion recognition with a hybrid attentional multimodal fusion framework[J]. IEEE Transactions on Affective Computing, 2023, 14(4): 2970-2981.
[82] YANG H H, WU J D, HU Z X, et al. Real-time driver cognitive workload recognition: attention-enabled learning with multimodal information fusion[J]. IEEE Transactions on Industrial Electronics, 2024, 71(5): 4999-5009.
[83] ZHAO J F, MAO X, CHEN L J. Speech emotion recognition using deep 1D & 2D CNN LSTM networks[J]. Biomedical Signal Processing and Control, 2019, 47: 312-323.
[84] FALAHZADEH M R, FARSA E Z, HARIMI A, et al. 3D convolutional neural network for speech emotion recognition with its realization on Intel CPU and NVIDIA GPU[J]. IEEE Access, 2022, 10: 112460-112471.
[85] BEHERA A, WHARTON Z, LIU Y H, et al. Regional attention network (RAN) for head pose and fine-grained gesture recognition[J]. IEEE Transactions on Affective Computing, 2023, 14(1): 549-562.
[86] FANG W J, TANG L R, PAN J H. AGL-net: an efficient neural network for EEG-based driver fatigue detection[J]. Journal of Integrative Neuroscience, 2023, 22(6): 146.
[87] DU X B, MA C X, ZHANG G H, et al. An efficient LSTM network for emotion recognition from multichannel EEG signals[J]. IEEE Transactions on Affective Computing, 2022, 13(3): 1528-1540.
[88] FU R, HUANG T, LI M Y, et al. A multimodal deep neural network for prediction of the driver’s focus of attention based on anthropomorphic attention mechanism and prior knowledge[J]. Expert Systems with Applications, 2023, 214: 119157.
[89] ZHOU H S, DU J, ZHANG Y Y, et al. Information fusion in attention networks using adaptive and multi-level factorized bilinear pooling for audio-visual emotion recognition[J]. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2021, 29: 2617-2629.
[90] LI S Z, XING X F, FAN W Q, et al. Spatiotemporal and frequential cascaded attention networks for speech emotion recognition[J]. Neurocomputing, 2021, 448: 238-248.
[91] GAN C Q, WANG K X, ZHU Q Y, et al. Speech emotion recognition via multiple fusion under spatial—temporal parallel network[J]. Neurocomputing, 2023, 555: 126623.
[92] LI W B, ZENG G Z, ZHANG J C, et al. CogEmoNet: a cognitive-feature-augmented driver emotion recognition model for smart cockpit[J]. IEEE Transactions on Computational Social Systems, 2022, 9(3): 667-678.
[93] HU H X, ZHANG L F, YANG T J, et al. A lightweight two-stream model for driver emotion recognition[J]. Journal of Physics: Conference Series, 2022, 2400(1): 012002.
[94] LIU S Q, WANG Z Y, AN Y L, et al. EEG emotion recognition based on the attention mechanism and pre-trained convolution capsule network[J]. Knowledge-Based Systems, 2023, 265: 110372.
[95] GURSESLI M C, LOMBARDI S, DURADONI M, et al. Facial emotion recognition (FER) through custom lightweight CNN model: performance evaluation in public datasets[J]. IEEE Access, 2024, 12: 45543-45559.
[96] LI C. Robotic emotion recognition using two-level features fusion in audio signals of speech[J]. IEEE Sensors Journal, 2022, 22(18): 17447-17454.
[97] LI X L, LIN H, DU J Z, et al. Computer vision-based driver fatigue detection framework with personalization threshold and multi-feature fusion[J]. Signal, Image and Video Processing, 2024, 18(1): 505-514.
[98] 张虎成, 李雷孝, 刘东江. 多模态数据融合研究综述[J]. 计算机科学与探索, 2024, 18(10): 2501-2520.
ZHANG H C, LI L X, LIU D J. Survey of multimodal data fusion research[J]. Journal of Frontiers of Computer Science and Technology, 2024, 18(10): 2501-2520.
[99] SAYED H M, ELDEEB H E, TAIE S A. Multimodal data fusion architectures in Audiovisual speech recognition[C]//Proceedings of the 2023 World Conference on Information Systems and Technologies. Cham: Springer, 2023: 655-667.
[100] HIEIDA C, YAMAMOTO T, KUBO T, et al. Negative emotion recognition using multimodal physiological signals for advanced driver assistance systems[J]. Artificial Life and Robotics, 2023, 28(2): 388-393.
[101] DING T Q, ZHANG K X, GAO S, et al. A multimodal driver anger recognition method based on context-awareness[J]. IEEE Access, 2024, 12: 118533-118550.
[102] MOU L T, ZHOU C, ZHAO P F, et al. Driver stress detection via multimodal fusion using attention-based CNN-LSTM[J]. Expert Systems with Applications, 2021, 173: 114693.
[103] MOU L T, ZHOU C, XIE P T, et al. Isotropic self-supervised learning for driver drowsiness detection with attention-based multimodal fusion[J]. IEEE Transactions on Multimedia, 2021, 25: 529-542.
[104] 张保梅. 数据级与特征级上的数据融合方法研究[D]. 兰州: 兰州理工大学, 2005.
ZHANG B M. Research on data fusion method at data level and feature level[D]. Lanzhou: Lanzhou University of Technology, 2005.
[105] OH G, JEONG E, KIM R C, et al. Multimodal data collection system for driver emotion recognition based on self-reporting in real-world driving[J]. Sensors, 2022, 22(12): 4402.
[106] SUN W C, LIU Y W, LI S W, et al. Research on driver’s anger recognition method based on multimodal data fusion[J]. Traffic Injury Prevention, 2024, 25(3): 354-363.
[107] HO N H, YANG H J, KIM S H, et al. Multimodal approach of speech emotion recognition using multi-level multi-head fusion attention-based recurrent neural network[J]. IEEE Access, 2020, 8: 61672-61686.
[108] LI W B, TAN R C, XING Y, et al. A multimodal psychological, physiological and behavioural dataset for human emotions in driving tasks[J]. Scientific Data, 2022, 9(1): 481.
[109] ANGKITITRAKUL P, PETRACCA M, SATHYANARAYANA A, et al. UTDrive: driver behavior and speech interactive systems for in-vehicle environments[C]//Proceedings of the 2007 IEEE Intelligent Vehicles Symposium. Piscataway: IEEE, 2007: 566-569.
[110] XIANG G L, YAO S, DENG H W, et al. A multi-modal driver emotion dataset and study: including facial expressions and synchronized physiological signals[J]. Engineering Applications of Artificial Intelligence, 2024, 130: 107772.
[111] MARCANTONI I, BARCHIESI G, BARCHIESI S, et al. Identification and classification of driving-related stress using electrocardiogram and skin conductance signals[C]//Proceedings of the 2022 IEEE International Symposium on Medical Measurements and Applications. Piscataway: IEEE, 2022: 1-6.
[112] OTHMAN W, KASHEVNIK A, ALI A, et al. DriverMVT: in-cabin dataset for driver monitoring including video and vehicle telemetry information[J]. Data, 2022, 7(5): 62.