基于交叉注意力的点击率预测模型

doi:10.3778/j.issn.1002-8331.2311-0420

摘要/Abstract

摘要： 有效特征的挖掘是点击率预测的关键，针对点击率预测模型对局部组合特征与全局特征间的信息交流考虑不充分，对组合特征的重要度表达不准确的问题，提出一种基于交叉注意力的点击率预测模型（CANN），通过交叉注意力机制突出组合特征与全局特征间的联系和交流，以充分挖掘有效特征。通过全局平均池化分别得到每个特征的特征值并拼接成全局特征；采用轴加权融合的方式来捕获组合特征；通过交叉注意力机制实现全局特征与组合特征交叉，得到组合特征的权重来表达其重要性，并将加权的组合特征融合到全局特征，以提高信息交流；通过多层感知机学习，得到点击率预测值。在两个公开和真实的数据集上的实验结果验证了该模型的有效性。

关键词: 点击率预测, 交叉注意力, 特征交互, 神经网络

Abstract: Mining effective features is the key for click through rate prediction. To address the problem of insufficient consideration of information exchange between local combined features and global feature, and inaccurate expression of the importance for the combined features in click through rate prediction models, a click through rate model based on cross attention (CANN) is proposed to emerge out the connection and communication between combined features and global feature by using cross attention mechanism to fully mine effective features. Firstly, each feature eigenvalue is respectively obtained by global average pooling, and all eigenvalues are concatenated to form a global feature. Then, the combination features are captured by axis weighted fusion. Next, the cross operation between the global feature and the combination features is performed by the cross attention mechanism, the weights of the combination features is obtained to express importance, and the weighted combination features are fused to the global feature to improve information exchange. Finally, the click through rate prediction value is obtained after learning by the multi-layer perceptron. Experiments on two public and real datasets verify the effectiveness of the model.

Key words: click through rate prediction, cross attention, feature interaction, neural network

何李杰, 高茂庭. 基于交叉注意力的点击率预测模型[J]. 计算机工程与应用, 2025, 61(7): 353-360.

HE Lijie, GAO Maoting. Click Through Rate Prediction Model Based on Cross Attention[J]. Computer Engineering and Applications, 2025, 61(7): 353-360.

参考文献

[1] CHEN B, WANG Y C, LIU Z R, et al. Enhancing explicit and implicit feature interactions via information sharing for parallel deep CTR models[C]//Proceedings of the 30th ACM International Conference on Information & Knowledge Management. New York: ACM, 2021: 3757-3766.
[2] LIU B, ZHU C X, LI G L, et al. AutoFIS: automatic feature interaction selection in factorization models for click-through rate prediction[C]//Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. New York: ACM, 2020: 2636-2645.
[3] MCMAHAN H B, HOLT G, SCULLEY D, et al. Ad click prediction[C]//Proceedings of the 19th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. New York: ACM, 2013: 1222-1230.
[4] RICHARDSON M, DOMINOWSKA E, RAGNO R, et al. Predicting clicks[C]//Proceedings of the 16th International Conference on World Wide Web. New York: ACM, 2007: 521-530.
[5] RENDLE S. Factorization machines[C]//Proceedings of the 2010 IEEE International Conference on Data Mining. Piscataway: IEEE, 2010: 995-1000.
[6] JUAN Y, ZHUANG Y, CHIN W S, et al. Field-aware factorization machines for CTR prediction[C]//Proceedings of the 10th ACM Conference on Recommender Systems. New York: ACM, 2016: 43-50.
[7] MU R H. A survey of recommender systems based on deep learning[J]. IEEE Access, 2018, 6: 69009-69022.
[8] COVINGTON P, ADAMS J, SARGIN E, et al. Deep neural networks for YouTube recommendations[C]//Proceedings of the 10th ACM Conference on Recommender Systems. New York: ACM, 2016: 191-198.
[9] CHENG H T, KOC L, HARMSEN J, et al. Wide & deep learning for recommender systems[C]//Proceedings of the 1st Workshop on Deep Learning for Recommender Systems. New York: ACM, 2016: 7-10.
[10] GUO H F, TANG R M, YE Y M, et al. DeepFM: a factorization-machine based neural network for CTR prediction[C]//Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, 2017: 1725-1731.
[11] HE X N, CHUA T S, HE X N, et al. Neural factorization machines for sparse predictive analytics[C]//Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval. New York: ACM, 2017: 355-364.
[12] QU Y R, CAI H, REN K, et al. Product-based neural networks for user response prediction[C]//Proceedings of the 2016 IEEE 16th International Conference on Data Mining. Piscataway: IEEE, 2016: 1149-1154.
[13] WANG R X, FU B, FU G, et al. Deep & cross network for ad click predictions[C]//Proceedings of the ADKDD’17. New York: ACM, 2017: 1-7.
[14] LIAN J X, ZHOU X H, ZHANG F Z, et al. xDeepFM: combining explicit and implicit feature interactions for recommender systems[C]//Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. New York: ACM, 2018: 1754-1763.
[15] 陈彬, 张荣梅, 张琦. DCFM: 基于深度学习的混合推荐模型[J]. 计算机工程与应用, 2021, 57(3): 150-155.
CHEN B, ZHANG R M, ZHANG Q. DCFM: hybrid recommendation model based on deep learning[J]. Computer Engineering and Applications, 2021, 57(3): 150-155.
[16] VASWANI A, SHAZEER N, Parmar N, et al. Attention is all you need[C]//Advances in Neural Information Processing Systems, 2017.
[17] CHENG Y, XUE Y B, CHENG Y, et al. Looking at CTR prediction again: is attention all you need?[C]//Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval. New York: ACM, 2021: 1279-1287.
[18] XIAO J, YE H, HE X N, et al. Attentional factorization machines: learning the weight of feature interactions via attention networks[C]//Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, 2017: 3119-3125.
[19] TAO Z L, WANG X, HE X N, et al. HoAFM: a high-order attentive factorization machine for CTR prediction[J]. Information Processing & Management, 2020, 57(6): 102076.
[20] SONG W P, SHI C C, XIAO Z P, et al. AutoInt: automatic feature interaction learning via self-attentive neural networks[C]//Proceedings of the 28th ACM International Conference on Information and Knowledge Management. New York: ACM, 2019: 1161-1170.
[21] LI X, CHEN S W, DONG J, et al. Decision-making context interaction network for click-through rate prediction[J]. Proceedings of the AAAI Conference on Artificial Intelligence, 2023, 37(4): 5195-5202.
[22] LIN H, CHENG X, WU X, et al. CAT: cross attention in vision transformer[C]//Proceedings of the 2022 IEEE International Conference on Multimedia and Expo, 2022: 1-6.
[23] JAEGLE A, BORGEAUD S, ALAYRAC J B, et al. Perceiver IO: a general architecture for structured inputs & outputs[J]. arXiv:2107.14795, 2021.
[24] HUANG T W, ZHANG Z Q, ZHANG J L, et al. FiBiNET: combining feature importance and bilinear feature interaction for click-through rate prediction[C]//Proceedings of the 13th ACM Conference on Recommender Systems. New York: ACM, 2019: 169-177.
[25] WANG Z Q, SHE Q Y, ZHANG J L, et al. MaskNet: introducing feature-wise multiplication to CTR ranking models by instance-guided mask[J]. arXiv:2102.07619, 2021.