Abstract: This paper constructs a multimodal biometric fusion model based on deep neural networks and combines several fusion strategies at different levels, such as pixel, feature, and score. The pixel level employs the three fusion strategies of spatial, channel, and intensity. The feature level builds first-order dependencies between modes by back-propagating the overall optimized modal specific branch and joint representation, and the score level completes the matching score fusion using both Rank1-based evaluation and modal-based evaluation methods. Besides, virtual homogeneous multimodal datasets are constructed by simulating real-world data. The experimental results show that the pixel level fusion method has limited improvement and it is difficult to improve data differentiation. The feature level fusion method combines image features with semantic features, which improves 2.2 percentage points when compared to the unimodal algorithm. The score level fusion method offers 3.5 percentage points improvement over the unimodal algorithm, with the optimal accuracy up to 99.6%. The generalizability and accuracy of the proposed multimodal biometric fusion model are significantly increased.

Key words: image processing, convolution neural network, multimodal, biometric retrieval

摘要： 面对公安实战中获取的低质量生物特征数据，单模态生物特征识别技术的精度并不理想，现有的多模态融合算法存在融合层次单一、泛化性不强等问题，深度神经网络的发展为其提供了有效的解决途径。构建基于深度神经网络的多模态生物特征融合模型，将像素层、特征层、分数层等不同层次的融合方法统一到融合模型中，在像素层采用空间、通道和强度融合三种策略；在特征层通过反向传播整体优化模态专用分支与联合表示层，构建模态之间一阶依赖关系；在分数层使用基于Rank1评价和基于模态评价两种方法完成匹配分数融合。模拟实战数据构建虚拟同源多模态数据集进行模型验证。实验结果表明，多模态像素层融合方法提升效果有限，难以增强数据的区分度；多模态特征层融合方法相比单模态算法提升2.2个百分点；分数层融合方法相比单模态算法提升3.5个百分点，最佳检索精度可达99.6%。基于深度学习方法提出的多模态生物特征融合模型极大地提高了模型的泛化性和检索精度。

关键词: 图像处理, 卷积神经网络, 多模态, 生物特征检索