Abstract: Expression recognition is becoming one of the latest challenges in computer vision and pattern recognition in recent years.In this paper，a computer vision system including both facial feature extraction and expression recognition is proposed.Facial motion features tracking and extraction are performed on the emotional speaking video first.Different with previous approach， segment the extracted feature vectors stream into two classifications.One is expression feature vector stream.The other is the visual speech one.Then，based on CHMM（Coupled Hidden Markov Model），which keeps the state asynchrony of the two observation sequences while preserving their natural correlation through the process，expression recognition is performed.The experimental results show that the coupled HMM outperforms the multi-stream HMM in video based emotion recognition.