Click-Through Rate Prediction Model of Enhanced High-Order Attentive Factorization Machine

doi:10.3778/j.issn.1002-8331.2106-0001

Abstract

Abstract: Modeling feature interactions is critical for predicting user click-through rate in recommendation system. This paper designs an enhanced high-order attention factorization machine （EHAFM） model, which is mainly composed of embedding layer, explicit feature interactions layer, and output layer. In each explicit feature interaction layer, the feature representation is updated by aggregating other feature representations, and for the problem that useless feature interactions interferes with the prediction, an enhanced bit-wise attention mechanism is proposed, it uses the projection matrix to expand the feature representation space, to enhance the learning ability of the attention matrix. In addition, the problem of insufficient model generalization ability is solved by fusing the information of multiple enhanced bit-wise attention heads. By stacking explicit feature interactions layers, the feature representations can be updated to any high-order, and finally the high-order feature interactions part and the first-order linear part are combined to predict the click-through rate. The EHAFM model is tested on two datasets, Criteo and Movielens-1M, the results show that compared with the benchmark model, there are 0.21% and 0.92% AUC improvements on the two datasets, respectively.

Key words: deep learning, explicit feature interactions, high-order factorization machine, attention mechanism

摘要： 特征交互的建模对于推荐系统中预测用户的点击率至关重要。设计了增强型高阶注意力因子分解机模型EHAFM（enhanced high-order attentive factorization machine），其主要由Embedding层、显式特征交互层、输出层构成。在每个显式特征交互层中，通过聚合其他特征的表示来更新特征的表示，并针对无用特征交互对预测产生干扰的问题，提出了增强型元素级注意力机制，利用投影矩阵拓展特征表示空间，以增强注意力矩阵的学习能力。通过融合多个增强型元素级注意力头的信息，以解决模型泛化能力不足问题。通过堆叠显式特征交互层可以将特征表示更新到任意高阶，将高阶特征交互部分与一阶线性部分结合进行点击率预测。EHAFM模型在Criteo、Movielens-1M两个数据集上进行实验，结果表明相较基准模型在两个数据集上分别有0.21%和0.92%的AUC提升。

关键词: 深度学习, 显式特征交互, 高阶因子分解机, 注意力机制

CHEN Yukang, LONG Huiyun, WU Yun, LIN Jian. Click-Through Rate Prediction Model of Enhanced High-Order Attentive Factorization Machine[J]. Computer Engineering and Applications, 2022, 58(24): 158-165.

陈育康, 龙慧云, 吴云, 林建. 增强型高阶注意力因子分解机点击率预测模型[J]. 计算机工程与应用, 2022, 58(24): 158-165.

References

[1] WANG X.A survey of online advertising click-through rate prediction models[C]//2020 IEEE International Conference on Information Technology，Big Data and Artificial Intelligence（ICIBA），2020：516-521.
[2] JIANG Z L，GAO S，DAI W.A CTR prediction approach for text advertising based on the SAE-LR deep neural network[J].Journal of Information Processing Systems，2017，13（5）：1052-1070.
[3] RICHARDSON M，DOMINOWSKA E，RAGNO R.Predicting clicks：estimating the click-through rate for newAds[C]//International World Wide Web Conference，Microsoft Research One Microsoft Way Redmond，2007：521-529.
[4] RENDLE S.Factorization machines[C]//The 10th IEEE International Conference on Data Mining，Sydney，Australia，14-17 December 2010：995-1000.
[5] XIAO J，YE H，HE X，et al.Attentional factorization machines：learning the weight of feature interactions via attention networks[J].arXiv：1708.04617，2017.
[6] TIAN Y，YANG G，WANG Z，et al.Apple detection during different growth stages in orchards using the improved YOLO-V3 model[J].Computers and Electronics in Agriculture，2019，157：417-426.
[7] YANG J，WANG M，ZHOU H，et al.Towards making the most of bert in neural machine translation[C]//Proceedings of the AAAI Conference on Artificial Intelligence，2020：9378-9385.
[8] CHENG H T，KOC L，HARMSEN J，et al.Wide & deep learning for recommender systems[C]//Proceedings of the 1st Workshop on Deep Learning for Recommender Systems，2016：7-10.
[9] GUO H，TANG R，YE Y，et al.DeepFM：a factorization machine based neural network for CTR prediction[J].arXiv：1703.04247，2017.
[10] 蒋兴渝，黄贤英，陈雨晶，等.基于特征增强聚合的融合广告点击率预测模型[J].计算机工程，2021（1）：312-320.
JIANG X Y，HUANG X Y，CHEN Y J，et al.Hybrid advertising click-through rate prediction model based on feature enhancement combination[J].Computer Engineering，2021（1）：312-320.
[11] WANG K，SHEN C，MA W.AdnFM：an attentive DenseNet based factorization machine for CTR prediction[J].arXiv：2012.10820，2020.
[12] WANG R，FU B，FU G，et al.Deep&cross network for ad click predictions[C]//Proceedings of the ADKDD’17，2017：1-7.
[13] LIAN J，ZHOU X，ZHANG F，et al.xdeepfm：combining explicit and implicit feature interactions for recommender systems[C]//Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining，2018：1754-1763.
[14] 陈彬，张荣梅，张琦.DCFM：基于深度学习的混合推荐模型[J].计算机工程与应用，2021，57（3）：150-155.
CHEN B，ZHANG R M，ZHANG Q.DCFM：hybrid recommendation model based on deep learning[J].Computer Engineeringand Applications，2021，57（3）：150-155.
[15] SONG W，SHI C，XIAO Z，et al.Autsoint：automatic feature interaction learning via self-attentive neural networks[C]//Proceedings of the 28th ACM International Conference on Information and Knowledge Management，2019：1161-1170.
[16] TAO Z L，WANG X，HE X N，et al.HoAFM：a high-order attentive factorization machine for CTR prediction[J].Information Prcessing & Management，2020，57.6：102076.
[17] HOFFMANN F，HOSSEINI B，REN Z，et al.Consistency of semi-supervised learning algorithms on graphs：probit and one-hot methods[J].Journal of Machine Learning Research，2020，21（186）：1-55.
[18] MIKOLOV T，CHEN K，CORRADO G，et al.Efficient estimation of word representations in vector space[J].arXiv：1301.3781，2013.
[19] BARKAN O，KOENIGSTEIN N.Item2vec：neural item embedding for collaborative filtering[C]//2016 IEEE 26th International Workshop on Machine Learning for Signal Processing（MLSP），2016：1-6.
[20] GRBOVIC M，CHENG H B.Real-time personalization using embeddings for search ranking at Airbnb[C]//Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining，2018：311-320.
[21] XIAO T，XU Y，YANG K，et al.The application of two-level attention models in deep convolutional neural network for fine-grained image classification[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2015：842-850.
[22] MARTINS A F T，ASTUDILLO R F.Fromsoftmax to sparse-max：a sparse model of attention and multi-labelclassification[J].arXiv：1602.02068，2016.
[23] VASWANI A，SHAZEER N，PARMAR N，et al.Attention is all you need[J].arXiv：1706.03762，2017.
[24] LU W，YU Y，CHANG Y，et al.A dual input-aware factorization machine for CTR prediction[C]//Proceedings of the 29th International Joint Conference on Artificial Intelligence，2020：3139-3145.
[25] KINGMA D P，BA J.Adam：a method for stochastic optimization[J].arXiv：1412.6980，2014.