大模型时代下的注意力机制优化综述

doi:10.3778/j.issn.1002-8331.2506-0314

摘要/Abstract

摘要： 自从注意力机制（attention mechanism）提出后，基于注意力机制的Transformer架构很快确立了大模型的核心地位，大语言模型迎来了新的发展方向，推动了自然语言处理、计算机视觉等众多领域取得了丰厚的成果。近年来，随着大模型迅猛发展，模型的参数规模持续增长，传统的Transformer架构已经难以满足庞大的模型训练要求。除了堆积算力之外，模型架构的调整以及对于注意力机制的进一步挖掘是解决这一挑战的有效途径，成为了研究的热点。介绍传统的Transformer架构以及近年来对于Transformer架构及其变体的研究现状，分析其核心自注意力机制的原理及其面临的瓶颈。分析总结近年来注意力模块的改进，以DeepSeek为例，探索其爆火背后基于Transformer的MoE架构和多头潜在注意力机制（multi-head latent attention，MLA）的核心技术路径。总结优化注意力机制的研究现状，并展望未来研究方向。

关键词: 大语言模型（LLM）, 注意力机制, 多头潜在注意力机制（MLA）, MoE架构

Abstract: Since the attention mechanism was proposed, the Transformer architecture based on the attention mechanism has quickly established the core position of large models. The large language model has ushered in a new development direction and promoted fruitful results in many fields such as natural language processing and computer vision. In recent years, with the rapid development of large models, the parameter scale of the model has continued to grow, and the traditional Transformer architecture has been difficult to meet the requirements of large model training. In addition to accumulating computing power, the adjustment of the model architecture and further exploration of the attention mechanism are effective ways to solve this challenge and have become a hot topic of research. This paper first introduces the traditional Transformer architecture and the current status of research on the Transformer architecture and its variants in recent years, and analyzes the principles of its core self-attention mechanism and the bottlenecks it faces. Subsequently, the improvements of the attention module in recent years are analyzed and summarized. Then, taking DeepSeek as an example, the core technical path of the Transformer-based MoE architecture and the multi-head latent attention mechanism (MLA) behind its explosion are explored. Finally, the current status of research on optimizing the attention mechanism is summarized and future research directions are prospected.

Key words: large language model (LLM), attention mechanism, multi-head latent attention mechanism (MLA), MoE architecture

史登辉, 奚雪峰, 崔志明, 朱润, 王坚. 大模型时代下的注意力机制优化综述[J]. 计算机工程与应用, 2026, 62(8): 34-47.

SHI Denghui, XI Xuefeng, CUI Zhiming, ZHU Run, WANG Jian. Review of Attention Mechanism in Era of Large Language Models[J]. Computer Engineering and Applications, 2026, 62(8): 34-47.

参考文献

[1] MNIH V，HEESS N，GRAVES A，et al.Recurrent models of visual attention[C]//Advances in Neural Information Processing Systems 27，2014：2204-2212.
[2] GRAVES A.Long short-term memory[M]//Supervised sequence labelling with recurrent neural networks.Berlin，Heidelberg：Springer，2012：37-45.
[3] BAHDANAU D，CHO K，BENGIO Y.Neural machine translation by jointly learning to align and translate[EB/OL].[2025-05-07].https：//arxiv.org/abs/1409.0473.
[4] VASWANI A，SHAZEER N，PARMAR N，et al.Attention is all you need[C]//Advances in Neural Information Processing Systems 30，2017：5998-6008.
[5] YAO C W，CAI D，BU J J，et al.Pre-training the deep generative models with adaptive hyperparameter optimization[J].Neurocomputing，2017，247：144-155.
[6] DEVLIN J，CHANG M W，LEE K，et al.BERT：pre-training of deep bidirectional transformers for language understanding[C]//Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics：Human Language Technologies，2019：4171-4186.
[7] WANG Y L，DENG Y J，ZHENG Y J，et al.Vision transformers for image classification：a comparative survey[J].Technologies，2025，13（1）：32.
[8] LIU Z，LIN Y T，CAO Y，et al.Swin transformer：hierarchical vision Transformer using shifted windows[C]//Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision.Piscataway：IEEE，2022：9992-10002.
[9] PINASTHIKA K，LAKSONO B S P，IRSAL R B P，et al.SparseSwin：swin Transformer with sparse Transformer block[J].Neurocomputing，2024，580：127433.
[10] LIU H T，LI C Y，WU Q Y，et al.?Visual instruction tuning[C]//Advances in Neural Information Processing Systems 36，2023：34892-34916.
[11] KELES F D，WIJEWARDENA P M，HEGDE C.On the computational complexity of self-attention[C]//Proceedings of the 34th International Conference on Algorithmic Learning Theory，2023：597-619.
[12] BENGIO Y，SIMARD P，FRASCONI P.Learning long-term dependencies with gradient descent is difficult[J].IEEE Transactions on Neural Networks，1994，5（2）：157-166.
[13] RADFORD A，WU J，CHILD R，et al.Language models are unsupervised multitask learners[J].OpenAI Blog，2019，1（8）：9.
[14] BROWN T B，MANN B，RYDER N，et al.Language models are few-shot learners[C]//Advances in Neural Information Processing Systems 33，2020：1877-1901.
[15] WANG S S，HU M Z，LI Q，et al.Capabilities of GPT-5 on multimodal medical reasoning[EB/OL].[2025-05-09].https：//arxiv.org/abs/2508.08224.
[16] NICOLSON A，PALIWAL K K.Masked multi-head self-attention for causal speech enhancement[J].Speech Communication，2020，125：80-96.
[17] LU Z L.Mechanisms of attention：psychophysics，cognitive psychology，and cognitive neuroscience[J].The Japanese Journal of Psychonomic Science，2008，27（1）：38-45.
[18] CORBETTA M，SHULMAN G L.Control of goal-directed and stimulus-driven attention in the brain[J].Nature Reviews Neuroscience，2002，3（3）：201-215.
[19] PETERSEN S E，POSNER M I.The attention system of the human brain：20 years after[J].Annual Review of Neuroscience，2012，35：73-89.
[20] DU Y Z，CHEN X，FU Y Q.Multiscale transformers and multi-attention mechanism networks for pathological nuclei segmentation[J].Scientific Reports，2025，15：12549.
[21] DEGUCHI H，TAMURA A，NINOMIYA T.Dependency-based self-attention for Transformer NMT[C]//Proceedings of the International Conference on Recent Advances in Natural Language Processing，2019：239-246.
[22] HE Y F，LIU J，WU W J，et al.ZipCache：accurate and efficient KV cache quantization with salient token identification[C]//Advances in Neural Information Processing Systems 37，2024：68287-68307.
[23] GRATTAFIORI A，DUBEY A，JAUHRI A，et al.The Llama 3 herd of models[EB/OL].[2025?05?12].https：//arxiv.org/abs/2407.21783.
[24] OPENAI，ACHIAM J，ADLER S，et al.GPT-4 technical report[EB/OL].[2025-05-12].https：//arxiv.org/abs/2303.08774.
[25] YUAN Z H，SHANG Y Z，ZHOU Y，et al.LLM inference unveiled：survey and roofline model insights[EB/OL].[2025-05-12].https：//arxiv.org/abs/2402.16363.
[26] QIN R Y，LI Z M，HE W R，et al.Mooncake：a KVCache-centric disaggregated architecture for LLM serving[EB/OL].[2025-05-13].https：//arxiv.org/abs/2407.00079.
[27] HU J C，LI H Y，ZHANG Y M，et al.Multi-matrix factorization attention[EB/OL].[2025-05-13].https：//arxiv.org/abs/2412.19255.
[28] TANG Y H，HU Z C，CHENG K，et al.Round attention：a novel round-level attention mechanism to accelerate LLM inference[EB/OL].[2025-05-14].https：//arxiv.org/abs/2502.
15294.
[29] FU T Y，HUANG H F，NING X F，et al.MoA：mixture of sparse attention for automatic large language model compression[EB/OL].[2025-05-14].https：//arxiv.org/abs/2406.
14909.
[30] FEDUS W，ZOPH B，SHAZEER N.Switch Transformers：scaling to trillion parameter models with simple and efficient sparsity[J].2022，23（1）：5232-5270.
[31] AINSLIE J，LEE-THORP J，DE JONG M，et al.GQA：training generalized multi-query Transformer models from multi-head checkpoints[EB/OL].[2025-05-14].https：//arxiv.org/abs/2305.13245.
[32] SHAZEER N.Fast Transformer decoding：one write-head is all you need[EB/OL].[2025?05?18].https：//arxiv.org/abs/1911.02150.
[33] LIU A X，FENG B，XUE B，et al.DeepSeek-V3 technical report[EB/OL].[2025-05-18].https：//arxiv.org/abs/2412.19437.
[34] DAO T，FU D Y，ERMON S，et al.FlashAttention：fast and memory-efficient exact attention with IO-awareness[EB/OL].[2025-05-18].https：//arxiv.org/abs/2205.14135.
[35] LIU H，ZAHARIA M，ABBEEL P.Ring attention with blockwise Transformers for near-infinite context[EB/OL].[2025-05-19].https：//arxiv.org/abs/2310.01889.
[36] YOON B，LEE A H，KIM J，et al.Exploring attention sparsity to accelerate transformer training on GPUs[J].IEEE Access，2024，12：131373-131384.
[37] ZHANG Z Y，XIAO C J，QIN Q，et al.Exploring the benefit of activation sparsity in pre-training[EB/OL].[2025-05-20].https：//arxiv.org/abs/2410.03440.
[38] BULATOV A，KURATOV Y，KAPUSHEV Y，et al.Beyond attention：breaking the limits of Transformer context length with recurrent memory[C]//Proceedings of the 38th AAAI Conference on Artificial Intelligence，2024：17700-17708.
[39] HUANG J H，BAO Y W.Rethinking transformers for efficiency and scalability[J].Available at SSRN 5161897，2025.
[40] MIJWIL M M，KAPAYA S.Global adoption of artificial intelligence[J].Innovations in Optimization and Machine Learning，2025：399.
[41] LI X Y，MENG Y X，ZHOU M X，et al.SAC：accelerating and structuring self-attention via sparse adaptive connection[C]//Advances in Neural Information Processing Systems 33，2020：16997-17008.
[42] ARAR M，SHAMIR A，BERMANO A H.Learned queries for efficient local attention[C]//Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition.Piscataway：IEEE，2022：10831-10842.
[43] TOUVRON H，MARTIN L，STONE K，et al.Llama 2：open foundation and fine-tuned chat models[EB/OL].[2025-05-22].https：//arxiv.org/abs/2307.09288.
[44] GLM T，XU B，ZENG A H，et al.ChatGLM：a family of large language models from GLM-130B to GLM-4 all tools[EB/OL].[2025-05-22].https：//arxiv.org/abs/2406.12793.
[45] CHEN Y A，ZHANG C，GAO X T，et al.Optimised grouped-query attention mechanism for Transformers[EB/OL].[2025-05-23].https：//arxiv.org/abs/2406.14963.
[46] DEROY A，MAITY S.Code generation and algorithmic problem solving using llama 3.1 405B[EB/OL].[2025-05-24].https：//arxiv.org/abs/2409.19027.
[47] 曾建勋.DeepSeek开源的力量[J].农业图书情报学报，2025，37（2）：100-101.
ZENG J X.The power of DeepSeek open source[J].Journal of Library and Information Science in Agriculture，2025，37（2）：100-101.
[48] KAPLAN J，MCCANDLISH S，HENIGHAN T，et al.Scaling laws for neural language models[EB/OL].[2025-05-25].https：//arxiv.org/abs/2001.08361.
[49] RAFFEL C，SHAZEER N，ROBERTS A，et al.Exploring the limits of transfer learning with a unified text-to-text transformer[J].Journal of Machine Learning Research，2020，21（140）：1-67.
[50] RAE J W，BORGEAUD S，CAI T，et al.Scaling language models：methods，analysis & insights from training gopher[EB/OL].[2025-05-25].https：//arxiv.org/abs/2112.11446.
[51] CHOWDHERY A，NARANG S，DEVLIN J，et al.PaLM：scaling language modeling with pathways[J].Journal of Machine Learning Research，2023，24（240）：1-113.
[52] JACOBS R A，JORDAN M I，NOWLAN S J，et al.Adaptive mixtures of local experts[J].Neural Computation，1991，3（1）：79-87.
[53] SHAZEER N，MIRHOSEINI A，MAZIARZ K，et al.Outrageously large neural networks：the sparsely-gated mixture-of-experts layer[EB/OL].[2025-05-26].https：//arxiv.org/abs/1701.06538.
[54] KONG R，LI Y C，WANG W J，et al.Serving MoE models on resource-constrained edge devices via dynamic expert swapping[J].IEEE Transactions on Computers，2025，74（8）：2799-2811.
[55] FAN Z W，SARKAR R，JIANG Z Y，et al.M3ViT：mixture-of-experts vision Transformer for efficient multi-task learning with modelaccelerator co-design[C]//Proceedings of the 36th International Conference on Neural Information Processing Systems，2022：28441-28457
[56] LI B，SHEN Y F，YANG J K，et al.Sparse mixture-of-experts are domain generalizable learners[EB/OL].[2025-05-27].https：//arxiv.org/abs/2206.04046.
[57] XUE F Z，SHI Z J，WEI F T，et al.Go wider instead of deeper[C]//Proceedings of the 36th AAAI Conference on Artificial Intelligence，2022：8779-8787.
[58] LIANG W X，YU L L，LUO L，et al.Mixture-of-Transformers：a sparse and scalable architecture for multi-modal foundation models[EB/OL].[2025-05-29].https：//arxiv.org/abs/2411.
04996.
[59] DAI Y，TANG D Y，LIU L X，et al.One model，multiple modalities：a sparsely activated approach for text，sound，image，video and code[EB/OL].[2025-05-29].https：//arxiv.org/abs/2205.06126.
[60] MUSTAFA B，RIQUELME C，PUIGCERVER J，et al.Multimodal contrastive learning with LIMoE：the language-image mixture of experts[C]//Advances in Neural Information Processing Systems 35，2022：9564-9576.
[61] MONDAL A，ANIRUDH R，SELVARAJ P.Multitask deep learning for the emulation and calibration of an agent-based malaria transmission model[J].PLoS Computational Biology，2025，21（7）：e1013330.
[62] PAN X L，LIN W X，ZHANG L，et al.FSMoE：a flexible and scalable training system for sparse mixture-of-experts models[C]//Proceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems.New York：ACM，2025：524-539.
[63] LI D C，LI X，WANG J，et al.Video recommendation with multi-gate mixture of experts soft actor critic[C]//Proceedings of the 43rd International ACM Conference on Research and Development in Information Retrieval.New York：ACM，2020：1553-1556.
[64] 曹泽麟，徐君，董振华，等.基于多任务学习的位置倾向性得分预测算法[J].计算机研究与发展，2023，60（1）：85-94.
CAO Z L，XU J，DONG Z H，et al.Prediction of the positional propensity scores based on multi task learning[J].Journal of Computer Research and Development，2023，60（1）：85-94.
[65] XU M W，CAI D Q，YIN W S，et al.Resource-efficient algorithms and systems of foundation models：a survey[J].ACM Computing Surveys，2025，57（5）：1-39.
[66] CAI W L，JIANG J Y，WANG F，et al.A survey on mixture of experts in large language models[J].IEEE Transactions on Knowledge and Data Engineering，2025，37（7）：3896-3915.
[67] 史宏志，赵健，赵雅倩，等.大模型时代的混合专家系统优化综述[J].计算机研究与发展，2025，62（5）：1164-1189.
SHI H Z，ZHAO J，ZHAO Y Q，et al.Survey on system optimization for mixture of experts in the era of large models[J].Journal of Computer Research and Development，2025，62（5）：1164-1189.
[68] JOHNSON A，GONZALEZ M，KIM D，et al.Improving large language model performance through compression and optimization[EB/OL].[2025-06-03].https：//d197for5662
m48.cloudfront.net/documents/publicationstatus/250935/preprint_pdf/8c8f5de0a8701e3d19c6da6240ef12d3.pdf.
[69] JI H S，SI Q Y，LIN Z，et al.Towards one-to-many visual question answering[C]//Findings of the Association for Computational Linguistics：EMNLP 2024，2024：16931-16943.
[70] DU N，HUANG Y P，DAI A M，et al.GLaM：efficient scaling of language models with mixture-of-experts[C]//Proceedings of the 39th International Conference on Machine Learning，2022：5547-5569.
[71] LANG J，HONG R P，XU J，et al.Biting off more than you can detect：retrieval-augmented multimodal experts for short video hate detection[C]//Proceedings of the ACM on Web Conference 2025.New York：ACM，2025：2763-2774.
[72] LIU A X，FENG B，WANG B，et al.DeepSeek-V2：a strong，economical，and efficient mixture-of-experts language model[EB/OL].[2025-06-07].https：//arxiv.org/abs/2405.04434.
[73] SU J L，AHMED M，LU Y，et al.RoFormer：enhanced Transformer with rotary position embedding[J].Neurocomputing，2024，568：127063.
[74] JI T，GUO B，WU Y B，et al.Towards economical inference：enabling DeepSeek’s multi-head latent attention in any Transformer-based LLMs[EB/OL].[2025-06-08].https：//arxiv.org/abs/2502.14837.
[75] SHEN K，GUO J L，TAN X，et al.A study on ReLU and softmax in Transformer[EB/OL].[2025-06-09].https：//arxiv.org/abs/2302.06461.
[76] FILIPEK A.Sparse query attention（SQA）：a computationally efficient attention mechanism with query heads reduction[EB/OL].[2025-06-11].https：//arxiv.org/abs/2510.01817.
[77] YANG M R，LI F X，XIAO T，et al.Multi-scale feature fusion using matrix decomposition for enhanced transformer performance on small-scale datasets[J].Journal of Circuits，Systems and Computers，2025，34（12）：2550291.
[78] HUANG Q Z，AN Z W，ZHUANG N，et al.Harder tasks need more experts：dynamic routing in MoE models[EB/OL].[2025-06-13].https：//arxiv.org/abs/2403.07652.
[79] BHATELE A，FEIZI S，HE S，et al.Loki：low-rank keys for efficient sparse attention[C]//Advances in Neural Information Processing Systems 37，2024：16692-16723.
[80] ZHOU L Q，PENG X H，CHEN M，et al.Unveiling patterns in spatial transcriptomics data：a novel approach utilizing graph attention autoencoder and multiscale deep subspace clustering network[J].GigaScience，2025，14：giae103.
[81] KHANAM R，HUSSAIN M，HILL R，et al.A comprehensive review of convolutional neural networks for defect detection in industrial applications[J].IEEE Access，2024，12：94250-94295.
[82] ZHANG R，MENG Q，ZHU R C，et al.Monte Carlo neural PDE solver for learning PDEs via probabilistic representation[J].IEEE Transactions on Pattern Analysis and Machine Intelligence，2025，47（6）：5059-5075.
[83] MU S Y，LIN S.A comprehensive survey of mixture-of-experts：algorithms，theory，and applications[EB/OL].[2025-06-18].https：//arxiv.org/abs/2503.07137.