Music Emotion Recognition Fusion on CNN-BiLSTM and Self-Attention Model

doi:10.3778/j.issn.1002-8331.2206-0113

Abstract

Abstract: With the development of music science and technology, music emotion recognition has been widely practiced and applied in music recommendation, music psychotherapy, sound and light scene construction, and so on. This paper simulates the process of human beings expressing emotions in music. Aiming at the long-distance dependence and low training efficiency of long-term and short-term memory neural networks in music emotion recognition, a new network model CBSA（CNN BiLSTM Self Attention） is proposed in applying to music emotion recognition regression training. The model uses a two-dimensional convolution neural network to obtain the local key features of music emotion, uses bidirectional long-term and short-term memory neural network to extract the serialized music emotion information from the obtained local key features, and uses the self-attention model to dynamically adjust the weight of the serialized information to highlight the global key points of music emotion. The experimental results show that the CBSA model can shorten the training time of analyzing the data rules in music emotion information, and effectively improve the accuracy of music emotion recognition.

Key words: music emotion recognition, two-dimensional convolutional neural network, bidirectional long and short-term memory neural network, self-attention model

摘要： 随着音乐科技研究的不断深入，音乐情感识别已被广泛实践和应用在音乐推荐、音乐心理治疗、声光场景构建等方面。模拟人类感受音乐表现情感的过程，针对音乐情感识别中长短时记忆神经网络的长距离依赖和训练效率低的问题，提出一种新的网络模型CBSA（CNN BiLSTM self attention），应用于长距离音乐情感识别回归训练。模型使用二维卷积神经网络获取音乐情感局部关键特征，采用双向长短时记忆神经网络从获取的局部关键特征中提取序列化音乐情感信息，利用自注意力模型对获取的序列化信息进行动态权重调整，突出音乐情感全局关键点。实验结果表明，CBSA模型可缩短分析音乐情感信息中数据规律的训练时间，有效地提高音乐情感识别精确度。

关键词: 音乐情感识别, 二维卷积神经网络, 双向长短时记忆神经网络, 自注意力模型

ZHONG Zhipeng, WANG Hailong, SU Guibin, LIU Lin, PEI Dongmei. Music Emotion Recognition Fusion on CNN-BiLSTM and Self-Attention Model[J]. Computer Engineering and Applications, 2023, 59(3): 94-103.

钟智鹏, 王海龙, 苏贵斌, 柳林, 裴冬梅. 融合CNN-BiLSTM和自注意力模型的音乐情感识别[J]. 计算机工程与应用, 2023, 59(3): 94-103.

References

[1] 李伟，李子晋，邵曦.音频音乐与计算机的交融——音频音乐技术[M].上海：复旦大学出版社，2019.
LI W，LI Z J，SHAO X.Audio music and computer integration-audio music technology[M].Shanghai：Fudan University Press，2019.
[2] LIU X，CHEN Q，WU X，et al.CNN based music emotion classification[J].arXiv：1704.05665，2017.
[3] 冯鹏宇.一种融合GRU和注意力机制的音乐分类推荐方法[D].广州：广东工业大学，2021.
FENG P Y.A music classification recommendation method combining GRU and attention mechanism[D].Guangzhou：Guangdong University of Technology，2021.
[4] YANG Y H，LIN Y C，SU Y，et al.A regression approach to music emotion recognition[J].IEEE Transactions on Audio，Speech，and Language Processing，2008，16（2）：448-457.
[5] HAN B J，RHO S，DANNENBERG R B，et al.SMERS：music emotion recognition using support vector regression[C]//International Society for Music Information Retrieval Conference，2009.
[6] 康健，王海龙，苏贵斌，等.音乐情感识别研究综述[J].计算机工程与应用，2022，58（4）：64-72.
KANG J，WANG H L，SU G B，et al.Survey of music emotion recognition[J].Computer Engineering and Applications，2022，58（4）：64-72.
[7] COUTINHO E，WENINGER F，SCHULLER B，et al.The munich LSTM-RNN approach to the MediaEval 2014 “Emotion in Music” task[C]//Proceedings of the MediaEval Multimedia Benchmark Workshop，2014.
[8] LI X，XIANYU H，TIAN J，et al.A deep bidirectional long short-term memory based multi-scale approach for music dynamic emotion prediction[C]//2016 IEEE International Conference on Acoustics，Speech and Signal Processing（ICASSP），2016.
[9] KOH E，DUBNOV S.Comparison and analysis of deep audio embeddings for music emotion recognition[J].arXiv：2104.06517，2021.
[10] 唐霞，张晨曦，李江峰.基于深度学习的音乐情感识别[J].电脑知识与技术，2019，15（11）：232-237.
TANG X，ZHANG C X，LI J F.Music emotion recognition based on deep learning[J].Computer Knowledge and Technology，2019，15（11）：232-237.
[11] 缪裕青，邹巍，刘同来，等.基于参数迁移和卷积循环神经网络的语音情感识别[J].计算机工程与应用，2019，55（10）：135-140.
MIAO Y Q，ZOU W，LIU T L，et al.Speech emotion recognition model based on parameter transfer and convolutional recurrent neural network[J].Computer Engineering and Applications，2019，55（10）：135-140.
[12] HIZLISOY S，YILDIRIM S，TUFEKCI Z.Music emotion recognition using convolutional long short term memory deep neural networks[J].Engineering Science and Technology an International Journal，2020，24（3）：760-767.
[13] 林颖.基于XGBoost-CNN模型的图片识别分析[D].厦门：厦门大学，2018.
LIN Y.Image recognition and analysis based on XGBoost-CNN model[D].Xiamen：Xiamen University，2018.
[14] CHAKI S，DOSHI P，PATNAIK P，et al.Attentive RNNs for continuous-time emotion prediction in music clips[C]//National Conference on Artificial Intelligence，2020：36-46.
[15] SCHUSTER M，PALIWAL K K.Bidirectional recurrent neural networks[J].IEEE Transactions on Signal Processing，2002，45（11）：2673-2681．
[16] 万兴.基于可能性模糊聚类的离群点检测算法研究[D].焦作：河南理工大学，2017.
WAN X.Research on outlier detection based on possibilistic fuzzy clustering[D].Jiaozuo：Henan Polytechnic University，2017.
[17] SOLEYMANI M，CARO M N，SCHMIDT E M，et al. 1000 songs for emotional analysis of music[C]//Proceedings of the 2nd ACM International Workshop on Crowdsourcing for Multimedia，2013：1-6.
[18] ALAJANKI A，YANG Y H，SOLEYMANI M.Benchmarking music emotion recognition systems[J].PLoS One，2016：835-838.
[19] MCFEE B，HUMPHREY E J，BELLO J P.A software framework for musical data augmentation[C]//Proceedings of the 16th ISMIR Conference，2015，2015：248-254.
[20] SALAMON J，BELLO J P.Deep convolutional neural networks and data augmentation for environmental sound classification[J].IEEE Signal Processing Letters，2017，24（3）：279-283.
[21] 周迅.基于数据增强策略的有效音频分类方法[D].武汉：武汉科技大学，2020.
ZHOU X.An effective audio classification method based on data augmentation strategy[D].Wuhan：Wuhan University of Science and Technology，2020.
[22] EYBEN F，SCHERER K R，SCHULLER B W，et al.The geneva minimalistic acoustic parameter set（GeMAPS） for voice research and affective computing[J].IEEE Transactions on Affective Computing，2017，7（2）：1.
[23] WEI X，CUCCHIARINI C，HOUT R V，et al.Acoustic correlates of speech intelligibility：the usability of the eGeMAPS feature set for atypical speech[C]//8th ISCA Workshop on Speech and Language Technology in Education，2019.
[24] TIAN L，MOORE J，LAI C.Recognising emotions in spoken dialogue with hierarchically fused acoustic and lexical features[C]//2016 IEEE Workshop on Spoken Language Technology，2017.
[25] PARADA-CABALEIRO E，BATLINER A，SCHEDL M.An exploratory study on the acoustic musical properties to decrease self-perceived anxiety[J].International Journal of Environmental Research and Public Health，2022，19（2）：994.
[26] SOLEYMANI M，CARO M N，SCHMIDT E M，et al. 1000 songs database[C]//Proceedings of ACM International Workshop on Crowdsourcing for Multimedia，2014：4-7.
[27] RUSSO M，KRALJEVI? L，STELLA M，et al.Cochleogram-based approach for detecting perceived emotions in music[J].Information Processing & Management，2020，57（5）：102270.
[28] BATHIGAMA T，MADUSHIKA S.Multi-representational music emotion recognition using deep convolution neural networks[J].arXiv：1705.4549，2017.
[29] CHOWDHURY S，PRAHER V，WIDMER G.Tracing back music emotion predictions to sound sources and intuitive perceptual qualities[J].arXiv：2106.07787，2021.