Computer Engineering and Applications ›› 2022, Vol. 58 ›› Issue (9): 187-194.DOI: 10.3778/j.issn.1002-8331.2012-0264

• Pattern Recognition and Artificial Intelligence • Previous Articles     Next Articles

DFSMN-T:Mandarin Speech Recognition with Language Model Transformer

HU Zhangfang, JIAN Fang, TANG Shanshan, MING Ziping, JIANG Bowen   

  1. School of Optoelectronic Engineering, Chongqing University of Posts and Telecommunications, Chongqing 400065, China
  • Online:2022-05-01 Published:2022-05-01



  1. 重庆邮电大学 光电工程学院,重庆 400065

Abstract: An automatic speech recognition system consists of two parts:an acoustic model and a language model, but the traditional language model N-gram has problems such as ignoring the semantic similarity of words and having too large parameters, which limit the further reduction of the error rate of speech recognition characters. To address the above problems, this paper proposes a novel speech recognition system that uses Chinese syllables(pinyin) as intermediate characters, uses a deep feed-forward sequence memory neural network DFSMN as the acoustic model, performs the speech-to-Chinese syllable task, and then understands pinyin-to-Chinese characters as a translation task, and introduces Transformer as the language model. At the same time, it proposes a method to reduce the computational complexity of Transformer by introducing Hadamard matrix for filtering when calculating attention weights and discarding parameters below a threshold value, thus resulting in faster model decoding. Experiments on datasets such as Aishell-1 and Thchs30 show that compared to the DFSMN combined with the 3-gram model, the DFSMN and improved Transformer-based speech recognition system achieves a relative decrease of 3.2% in character error rate to 11.8% character error rate on the optimal model. Compared to the BLSTM model speech recognition system, its character error rate is relatively reduced by 7.1%, compared to the BLSTM model speech recognition system.

Key words: speech recognition, deep feedforward sequential memory networks(DFSMN), Transformer, Chinese syllables, Hadamard matrix

摘要: 自动语音识别系统由声学模型和语言模型两部分构成,但传统语言模型N-gram存在忽略词条语义相似性、参数过大等问题,限制了语音识别字符错误率的进一步降低。针对上述问题,提出一种新型的语音识别系统,以中文音节(拼音)作为中间字符,以深度前馈序列记忆神经网络DFSMN作为声学模型,执行语音转中文音节任务,进而将拼音转汉字理解成翻译任务,引入Transformer作为语言模型;同时提出一种减少Transformer计算复杂度的简易方法,在计算注意力权值时引入Hadamard矩阵进行滤波,对低于阈值的参数进行丢弃,使得模型解码速度更快。在Aishell-1、Thchs30等数据集上的实验表明,相较于DFSMN结合3-gram模型,基于DFSMN和改进Transformer的语音识别系统在最优模型上的字符错误率相对下降了3.2%,达到了11.8%的字符错误率;相较于BLSTM模型语音识别系统,其字符错误率相对下降了7.1%。

关键词: 语音识别, 深度前馈序列记忆神经网络(DFSMN), Transformer, 中文音节, Hadamard矩阵