Neural Recommendation Algorithm Using Combinations of Low and High-Order Features Based on Multi-Attention Mechanism

doi:10.3778/j.issn.1002-8331.2112-0570

Abstract

Abstract: Aimed at the limitations of factorization machine in extracting only low-order combinatorial features, a deep neural recommendation method based on multi-attention mechanism（DeepNRM） is proposed based on multi-attention mechanism. Firstly, the factorization machine and multi-layer feed-forward neural network are used to extract low-order and high-order combined features from sparse and dense features respectively. Then it uses the attention network and the multi-head self-attention mechanism to automatically select key features from the combination of low-order and high-order features. Finally, it mixes low-order and high-order combined features based on their importance to produce recommendations. The algorithm model is experimentally validated on the common datasets of MovieLens and Criteo, and the ablation and comparative experimental results show that compared with the baseline model, there are 1.964 percentage points and 0.773 percentage points AUC improvements on the two datasets, respectively.

Key words: factorization machine, recommendation system, deep neural network, multi-head self-attention mechanism, feature extraction

摘要： 针对因子分解机仅提取低阶组合特征的局限性，提出了一种基于多注意力机制融合低阶和高阶组合特征的深度神经推荐算法（deep neural recommendation method，DeepNRM）。分别运用因子分解机和多层前馈神经网络从稀疏及稠密特征中提取低阶和高阶组合特征；采用注意力网络和多头自注意力机制从低阶和高阶组合特征中自动选取关键特征；将低、高阶组合特征根据重要性进行融合共同进行推荐。算法模型在MovieLens和Criteo公共数据集上进行了实验验证，消融和对比实验结果表明，提出的算法模型与基准模型相比在AUC指标上分别有1.964个百分点和0.773个百分点的提升。

CUI Shaoguo, DU Xiao, YANG Zetian. Neural Recommendation Algorithm Using Combinations of Low and High-Order Features Based on Multi-Attention Mechanism[J]. Computer Engineering and Applications, 2023, 59(8): 192-199.

崔少国, 独潇, 杨泽田. 多注意力机制融合低高阶特征的神经推荐算法[J]. 计算机工程与应用, 2023, 59(8): 192-199.

References

[1] 项亮.推荐系统实践[M].北京：人民邮电出版社，2012.
XIANG L.Recommender system practices[M].Beijing：Publishing House of Posts and Telecommunications，2012.
[2] 黄立威，江碧涛，吕守业，等.基于深度学习的推荐系统研究综述[J].计算机学报，2018，41（7）：1619-1647.
HUANG L W，JIANG B T，LV S Y，et al.Survey on deep learning based recommender systems[J].Chinese Journal of Computers，2018，41（7）：1619-1647.
[3] 王瑞平，贾真，刘畅，等.基于DeepFM的深度兴趣因子分解机网络[J].计算机科学，2021，48（1）：226-232.
WANG R P，JIA Z，LIU C，et al.Deep interest factorization machine network based on DeepFM[J].Computer Science，2021，48（1）：226-232.
[4] 刘君良，李晓光.个性化推荐系统技术进展[J].计算机科学，2020，47（7）：47-55.
LIU J L，LI X G.Techniques for recommendation system：a survey[J].Computer Science，2020，47（7）：47-55.
[5] 王喆.深度学习推荐系统[M].北京：电子工业出版社，2020.
WANG Z.Deep learning recommender system[M].Beijing：Publishing House of Electronics Industry，2020.
[6] COVINGTON P，ADAMS J，SARGIN E.Deep neural networks for youtube recommendations[C]//Proceedings of the 10th ACM Conference on Recommender Systems，2016：191-198.
[7] CHENG H T，KOC L，HARMSEN J，et al.Wide & deep learning for recommender systems[C]//Proceedings of the 1st Workshop on Deep Learning for Recommender Systems，2016：7-10.
[8] GUO H，TANG R，YE Y，et al.DeepFM：a factorization-machine based neural network for CTR prediction[C]//Proceedings of the 26th International Joint Conference on Artificial Intelligence，2017：1725-1731.
[9] HE X，CHUA T S.Neural factorization machines for sparse predictive analytics[C]//Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval，2017：355-364.
[10] XIAO J，YE H，HE X，et al.Attentional factorization machines：learning the weight of feature interactions via attention networks[C]//Proceedings of the 26th International Joint Conference on Artificial Intelligence，2017：3119-3125.
[11] SONG W，SHI C，XIAO Z，et al.Autoint：automatic feature interaction learning via self-attentive neural networks[C]//Proceedings of the 28th ACM International Conference on Information and Knowledge Management，2019：1161-1170.
[12] RENDLE S.Factorization machines[C]//2010 IEEE International Conference on Data Mining，2010：995-1000.
[13] LUONG M T，PHAM H，MANNING C D.Effective approaches to attention-based neural machine translation[J].arXiv：1508.04025，2015.
[14] VASWANI A，SHAZEER N，PARMAR N，et al.Attention is all you need[C]//Advances in Neural Information Processing Systems，2017：5998-6008.
[15] HOFFMANN F，HOSSEINI B，REN Z，et al.Consistency of semi-supervised learning algorithms on graphs：probit and one-hot methods[J].Journal of Machine Learning Research，2020，21：7549-7603.
[16] ZHAO W X，MU S，HOU Y，et al.Recbole：towards a unified，comprehensive and efficient framework for recommendation algorithms[C]//Proceedings of the 30th ACM International Conference on Information & Knowledge Management，2021：4653-4664.
[17] QUEVEDO J R，MONTA?éS E，RANILLA J，et al.Ranked tag recommendation systems based on logistic regression[C]//International Conference on Hybrid Artificial Intelligence Systems.Berlin，Heidelberg：Springer，2010：237-244.
[18] WANG R，FU B，FU G，et al.Deep & cross network for ad click predictions[C]//Proceedings of the ADKDD’17，2017：1-7.
[19] 吴韦俊，李烨.神经因子分解机推荐模型改进研究[J].软件导刊，2020，19（4）：115-118.
WU W J，LI H.Research on improvement of recommendation model based on neural factorization machines[J].Software Guide，2020，19（4）：115-118.
[20] 陈彬，张荣梅，张琦.DCFM：基于深度学习的混合推荐模型[J].计算机工程与应用，2021，57（3）：150-155.
CHEN B，ZHANG R M，ZHANG Q.DCFM：hybrid recommendation model based on deep learning[J].Computer Engineering and Applications，2021，57（3）：150-155.
[21] KINGMA D，BA J.Adam：a method for stochastic optimization[J].arXiv：1412.6980，2014.
[22] SRIVASTAVA N，HINTON G，KRIZHEVSKY A，et al.Dropout：a simple way to prevent neural networks from overfitting[J].The Journal of Machine Learning Research，2014，15（1）：1929-1958.