Abstract: This paper proposes a modified triple stream asynchronous DBN model（VVA_AsyDBN）for audio visual emotion recognition, with the two visual feature streams, facial geometric features（GF） and facial active appearance model features（AAM）, synchronous at the state level, while they are asynchronous with the audio feature stream（Mel Filterbank Cepstrum Coefficients, MFCC） within controllable constraints. Emotion recognition experiments are carried out on the eNTERFACE’05 database, and results are compared with the traditional state synchronous Multi-Stream Hidden Markov Model（MSHMM）, as well as the asynchronous DBN model（T_AsyDBN） with two audio feature streams（MFCC and local prosodic features LP） and one visual feature stream. Results show that VVA_AsyDBN obtains the highest performance up to 75.61%, which is 12.50% higher than the visual only HMM, 2.32% higher than the MSHMM with MFCC, AAM and GF features, and 1.65% higher than the T_AsyDBN model with MFCC and LP features as well as AAM features.

Key words: audio visual fusion, Dynamic Bayesian Network（DBN）, Active Appearance Model（AAM）, asynchrony constraint

摘要： 提出了一个改进的三特征流听视觉融合异步动态贝叶斯网络情感模型（VVA_AsyDBN），采用面部几何特征（GF）和面部主动外观模型特征（AAM）作为两个视觉输入流，语音Mel倒谱特征（MFCC）作为听觉输入流，且视觉流的状态和听觉流的状态可以存在有约束的异步。在eNTERFACE’05听视觉情感数据库上进行了情感识别实验，并与传统的多流同步隐马尔可夫模型（MSHMM），以及具有两个听觉特征流（语音MFCC和局域韵律特征LP）和一个视觉特征流的听视觉异步DBN模型（T_AsyDBN）进行了比较。实验结果表明，VVA_AsyDBN获得了最高识别率75.61%，比视觉单流HMM提高了12.50%，比采用AAM、GF和MFCC特征的MSHMM提高了2.32%，比T_AsyDBN的最高识别率也提高了1.65%。

关键词: 听视觉融合, 动态贝叶斯网络, 主动外观模型（AAM）, 异步约束