Abstract: H.264/AVC provides 17 modes for intra prediction，and the optimal mode is selected by Rate Distortion Optimization （RDO） with high coding complexity.Although most of the current algorithms of intra prediction mode decision can reduce coding complexity greatly，there still exists high computational complexity.A new fast intra prediction mode decision algorithm based on BP neural network is proposed.The algorithm firstly judges the mode of the macroblock as either Intra16×16 or Intra4×4 mode，using the training result of the BP neural network with lots of video sequence.Then the optimal mode is decided from 17 modes by Pan Feng’s method.Experimental results show that the proposed algorithm can efficiently reduce the computational complexity up to 75%～83% compared to JM86 and 30%～70% compared to Pan Feng’s method，and can maintain the coding performance （the reduction of PSNR is only 0.01 dB on average）.

Key words: H.264, intra prediction, mode decision, BP network

摘要： H.264/AVC采用17种预测模式和率失真优化模型来进行帧内模式的选择，复杂度很高。目前的快速帧内模式选择算法能有效减少模式选择的复杂度，但还存在很大的不足。提出了一种基于BP神经网络的快速帧内模式选择算法。该算法使用BP神经网络对大量视频序列预先训练的结果对宏块进行预判，得出当前宏块采用Intra16×16或Intra4×4模式，然后使用Pan Feng的方法判断具体选择何种模式。实验结果表明，该算法在保证视频质量几乎不变（PSNR平均降低约0.01 dB）的同时，可以有效减少计算复杂度，与JM86相比减少75%～83%，与Pan Feng的方法相比减少30%～70%。

关键词: H.264, 帧内编码, 模式选择, BP神经网络