基于深度学习的多模态生物特征融合模型

doi:10.3778/j.issn.1002-8331.2205-0498

摘要/Abstract

摘要： 面对公安实战中获取的低质量生物特征数据，单模态生物特征识别技术的精度并不理想，现有的多模态融合算法存在融合层次单一、泛化性不强等问题，深度神经网络的发展为其提供了有效的解决途径。构建基于深度神经网络的多模态生物特征融合模型，将像素层、特征层、分数层等不同层次的融合方法统一到融合模型中，在像素层采用空间、通道和强度融合三种策略；在特征层通过反向传播整体优化模态专用分支与联合表示层，构建模态之间一阶依赖关系；在分数层使用基于Rank1评价和基于模态评价两种方法完成匹配分数融合。模拟实战数据构建虚拟同源多模态数据集进行模型验证。实验结果表明，多模态像素层融合方法提升效果有限，难以增强数据的区分度；多模态特征层融合方法相比单模态算法提升2.2个百分点；分数层融合方法相比单模态算法提升3.5个百分点，最佳检索精度可达99.6%。基于深度学习方法提出的多模态生物特征融合模型极大地提高了模型的泛化性和检索精度。

关键词: 图像处理, 卷积神经网络, 多模态, 生物特征检索

Abstract: This paper constructs a multimodal biometric fusion model based on deep neural networks and combines several fusion strategies at different levels, such as pixel, feature, and score. The pixel level employs the three fusion strategies of spatial, channel, and intensity. The feature level builds first-order dependencies between modes by back-propagating the overall optimized modal specific branch and joint representation, and the score level completes the matching score fusion using both Rank1-based evaluation and modal-based evaluation methods. Besides, virtual homogeneous multimodal datasets are constructed by simulating real-world data. The experimental results show that the pixel level fusion method has limited improvement and it is difficult to improve data differentiation. The feature level fusion method combines image features with semantic features, which improves 2.2 percentage points when compared to the unimodal algorithm. The score level fusion method offers 3.5 percentage points improvement over the unimodal algorithm, with the optimal accuracy up to 99.6%. The generalizability and accuracy of the proposed multimodal biometric fusion model are significantly increased.

Key words: image processing, convolution neural network, multimodal, biometric retrieval

李卓容, 唐云祁. 基于深度学习的多模态生物特征融合模型[J]. 计算机工程与应用, 2023, 59(7): 180-189.

LI Zhuorong, TANG Yunqi. Multimodal Biometric Fusion Model Based on Deep Learning[J]. Computer Engineering and Applications, 2023, 59(7): 180-189.

参考文献

[1] SINGH L，KHANNA M，GARG H.Multimodal biometric based on fusion of ridge features with minutiae features and face features[J].International Journal of Information System Modeling and Design（IJISMD），2020，11（1）：37-57.
[2] JAIN A，KUMAR A.Biometric recognition：an overview[J].Second Generation Biometrics：the Ethical，Legal and Social Context，2012（11）：49-79.
[3] 张永良，杨杰，吴鸿韬，等.基于扩展相位相关方法的滑动指纹序列拼接[J].上海交通大学学报，2006，40（3）：5.
ZHANG Y L，YANG J，WU H T，et al.Swipe fingerprint sequence mosaicking based on the extension of phase correlation[J].Journal of Shanghai Jiaotong University，2006，40（3）：5.
[4] HEO J，KONG S，ABIDI B，et al.Fusion of visual and thermal signatures with eyeglass removal for robust face recognition[C]//Proceedings of the 2004 Conference on Computer Vision and Pattern Recognition Workshop，2004：122-122.
[5] 宁雪丽，陈兆学.用于面诊的人脸全景图像拼接算法[J].中国医学物理学杂志，2020，37（4）：456-462.
NING X L，CHEN Z X.Face panoramic image mosaic algorithm for face diagnosis[J].Chinese Journal of Medical Physics，2020，37（4）：456-462.
[6] YAMAN D，EYIOKUR F I，EKENEL H K.Multimodal age and gender classification using ear and profile face images[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops（CVPRW），Long Beach，CA，USA，June 2019.
[7] RATTANI A，TISTARELLI M.Robust multi-modal and multi-unit feature level fusion of face and iris biometrics[C]//Proceedings of the International Conference on biometrics.Berlin，Heidelberg：Springer，2009：960-969.
[8] HU Q，WU C，CHI J，et al.Multi-level feature fusion facial expression recognition network[C]//Proceedings of the 2020 Chinese Control and Decision Conference（CCDC），2020：5267-5272.
[9] 孔俊.基于双层特征融合的生物识别[J].北华大学学报（自然科学版），2020，21（1）：110-117.
KONG J.Biometric identification based on two-layer feature fusion[J].Journal of Beihua University（Natural Science），2020，21（1）：110-117.
[10] MADANE M，THEPADE S.Score level fusion based Multimodal biometric identification using Thepade’s sorted ternary block truncation coding with variod proportion of iris，palmprint，left fingerprint & right fingerprint with asorted similarity measures & different Colorspaces[C]//Proceedings of the 2016 International Conference on Automatic Control and Dynamic Optimization Techniques（ICACDOT），2016：824-828.
[11] NANDAKUMAR K，CHEN Y，DASS S，et al.Likelihood ratio-based biometric score fusion[J].IEEE Transactions on Pattern Analysis and Machine Intelligence，2007，30（2）：342-347.
[12] 李洪安，杜卓明，李占利，等.基于双特征匹配层融合的步态识别方法[J].图学学报，2019，40（3）：441-446.
LI H A，DU Z　Ｍ，LI Z　Ｌ，et al.Research on gait recognition algorithm based on double features using the layer matching fusion method[J].Journal of Graphics，2019，40（3）：441-446.
[13] HO T，HULL J，SRIHARI S.Decision combination in multiple classifier systems[J].IEEE Transactions on Pattern Analysis And Machine Intelligence，1994，16（1）：66-75.
[14] FAUNDEZ M.Data fusion in biometrics[J].IEEE Aerospace and Electronic Systems Magazine，2005，20（1）：34-38.
[15] ROSS A，GOVINDARAJAN R.Feature level fusion of hand and face biometrics[C]//Proceedings of the Biometric Technology for Human Identification II.International Society for Optics and Photonics，2005：196-204.
[16] LIU C，WECHSLER H.A shape-and texture-based enhanced fisher classifier for face recognition[J].IEEE Transactions on Image Processing，2001，10（4）：598-608.
[17] YANG J，YANG J.Generalized K-L transform based combined feature extraction[J].Pattern Recognition，2002，35（1）：295-297.
[18] YANG J，YANG J，ZHANG D，et al.Feature fusion：parallel strategy vs.serial strategy[J].Pattern Recognition，2003，36（6）：1369-1381.
[19] HAGHIGHAT M，ABDEL-MOTTALEB M，ALHALABI W.Discriminant correlation analysis：real-time feature level fusion for multimodal biometric recognition[J].IEEE Transactions on Information Forensics and Security，2016，11（9）：1984-1996.
[20] SUN Q，ZENG S，LIU Y，et al.A new method of feature fusion and its application in image recognition[J].Pattern Recognition，2005，38（12）：2437-2448.
[21] DINCA L，HANCKE G.The fall of one，the rise of many：a survey on multi-biometric fusion methods[J].IEEE Access，2017，5：6247-6289.
[22] JOSHI T，DEY S，SAMANTA D.Multimodal biometrics：state of the art in fusion techniques[J].International Journal of Biometrics，2009，1（4）：393-417.
[23] NATARAJAN P，WU S，VITALADEVUNI S，et al.Multimodal feature fusion for robust event detection in web videos[C]//Proceedings of the 2012 IEEE Conference on Computer Vision and Pattern Recognition，2012：1298-1305.
[24] XIN Y，KONG L，LIU Z，et al.Multimodal feature-level fusion for biometrics identification system on IoMT platform[J].IEEE Access，2018，6：21418-21426.
[25] SANJEKAR P，PATIL J.Multimodal biometrics using fingerprint，palmprint，and iris with a combined fusion approach[J].International Journal of Computer Vision and Image Processing（IJCVIP），2019，9（4）：1-14.
[26] SUPREETHA G，KUMAR H，IMRAN M.Multimodal biometric verification system：evaluation of various score level fusion rules[C]//Proceedings of the 2019 IEEE International Conference on Electrical，Computer and Communication Technologies（ICECCT），2019：1-4.
[27] MEHROTRA H，RATTANI A，GUPTA P.Fusion of iris and fingerprint biometric for recognition[C]//Proceedings of International Conference on Signal and Image Processing，2006：1-6.
[28] WANG Y，TAN T，JAIN A.Combining face and iris biometrics for identity verification[C]//Proceedings of the International Conference on Audio-and Video-Based Biometric Person Authentication.Berlin，Heidelberg：Springer，2003：805-813.
[29] ARAVINTH J，VALARMATHY S.Multi classifier-based score level fusion of multi-modal biometric recognition and its application to remote biometrics authentication[J].The Imaging Science Journal，2016，64（1）：1-14.
[30] NANDAKUMAR K，JAIN A，ROSS A.Fusion in multibiometric identification systems：what about the missing data?[C]//Proceedings of the International Conference on Biometrics.Berlin，Heidelberg：Springer，2009：743-752.
[31] ZHU Z，HUANG G，DENG J，et al.WebFace260M：a benchmark unveiling the power of million-scale deep face recognition[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2021：10492-10502.
[32] SU K，YANG G，WU B，et al.Human identification using finger vein and ECG signals[J].Neurocomputing，2019，332：111-118.
[33] SNELICK R，ULUDAG U，MINK A，et al.Large-scale evaluation of multimodal biometric authentication using state-of-the-art systems[J].IEEE Transactions on Pattern Analysis and Machine Intelligence，2005，27（3）：450-455.