A-DDPG：多用户边缘计算系统的卸载研究

doi:10.3778/j.issn.1002-8331.2203-0461

摘要/Abstract

摘要： 为了降低多边缘服务器多用户系统中用户的总成本，结合深度确定性策略梯度（deep deterministic policy gradient，DDPG）、长短期记忆网络（LSTM）和注意力机制，提出了一种基于DDPG的深度强化学习卸载算法（A-DDPG）。该算法采用二进制卸载策略，并且将任务的延迟敏感性和服务器负载的有限性以及任务迁移考虑在内，自适应地卸载任务，以最大限度减少由延迟敏感型任务超时造成的总损失。考虑时延和能耗两个指标并设定了不同的权重值，解决因用户类型不同带来的不公平问题，制定了任务卸载问题以最小化所有任务完成时延和能量消耗的总成本，以目标服务器的选择和数据卸载量为学习目标。实验结果表明，A-DDPG算法具有良好的稳定性和收敛性，与DDPG算法和双延迟深度确定性策略梯度（twin delayed deep deterministic policy gradient，TD3）算法相比，A-DDPG算法的用户总成本分别降低了27%和26.66%，平均达到最优任务失败率的时间分别提前了57.14%和40%，其在奖励、总成本和任务失败率方面取得了较好的效果。

关键词: 移动边缘计算, 计算卸载, 深度确定性策略梯度（DDPG）, 资源分配

Abstract: In order to reduce the total cost of users in multi-user systems with multiple edge servers, a deep reinforcement learning offloading algorithm（A-DDPG） based on DDPG is proposed by combining deep deterministic policy gradient（DDPG）, long short term memory（LSTM） and attention mechanism, which uses binary offloading strategy and takes into account the latency sensitivity of tasks and the limited server load as well as task migration to adapt offload tasks to minimize the total loss caused by latency-sensitive task timeouts. Two metrics, latency and energy consumption, are considered and different weight values are set to address the unfairness caused by different user types, and the task offloading problem is formulated to minimize the total cost of all task completion latency and energy consumption, with the selection of target servers and the amount of data offloaded as learning objectives. The experimental results show that the A-DDPG algorithm has good stability and convergence, and the total user cost of the A-DDPG algorithm is reduced by 27% and 26.66% compared to the DDPG algorithm and the twin delayed deep deterministic policy gradient（TD3） algorithm respectively. It achieves better results in terms of reward, total cost and task failure rate, as the average time to reach the optimal task failure rate is 57.14% and 40% earlier, respectively.

Key words: mobile edge computing, computational offloading, , deep deterministic policy gradient（DDPG）, resource allocation

曹绍华, 姜佳佳, 陈舒, 詹子俊, 张卫山. A-DDPG：多用户边缘计算系统的卸载研究[J]. 计算机工程与应用, 2023, 59(1): 259-268.

CAO Shaohua, JIANG Jiajia, CHEN Shu, ZHAN Zijun, ZHANG Weishan. A-DDPG：Research on Offloading of Multi-User Edge Computing System[J]. Computer Engineering and Applications, 2023, 59(1): 259-268.

参考文献

[1] ZHANG J，ZHAOX.An overview of user-oriented computation offloading in mobile edge computing[C]//Proceedings of 2020 IEEE World Congress on Service，2020：75-76.
[2] HUANG L，FENG X，QIAN L，et al.Deep reinforcement learning-based task offloading and resource allocation for mobile edge computing[J].Machine Learning and Intelligent Communications，2018，251：33-42.
[3] PREMSANKAR G，GHADDAR B，MD FRANCESCO，et al.Efficient placement of edge computing devices for vehicular applications in smart cities[C]//Proceedings of IEEE/IFIP Networking Operations & Management Symposium，2018：1-9.
[4] MENG H，CHAO D，GUO Q.Deep reinforcement learning based task offloading algorithm for mobile-edge computing systems[C]//Proceedings of the 2019 4th International Conference，2019：90-94.
[5] WANG Y，GE H，FENG A，et al.Computation offloading strategy based on deep reinforcement learning in cloud-assisted mobile edge computing[C]//Proceedings of 2020 IEEE 5th International Conference on Cloud Computing and Big Data Analytics，2020：108-113.
[6] SONG S，FANG Z，ZHANG Z，et al.Semi-online comput-ational offloading by dueling deep-Q network for user behavior prediction[J].IEEE Access，2020，8：118192-118204.
[7] XU X，SHEN B，DING S，et al.Service offloading with deep Q-network for digital twinning empowered internet of vehicles in edge computing[J].IEEE Transactions on Industrial Informatics，2022，18（2）：1414-1423.
[8] CHEN X，JIAO L，LI W H et al.Efficient multi-user computation offloading for mobile edge cloud computing[J].ACM Transactions on Networking，2016，24（5）：2795-2808.
[9] LI N，JOSE-FERNAN M O，HERNANDEZ D V.Distrib-uted power control for interference-aware multi-user mobile edge computing：a game theory approach[J].IEEE Access，2018，6：36105-36114.
[10] ZHENG J，WU Y，ZHANG N，et al.Optimal power con-trol in ultra-dense small cell networks：A game-theoretic approach[J].IEEE Transactions on Wireless Communications，2017，16（7）：4139-4150.
[11] NING Z，SHAN Z，ZHENGG J，et al.QoE driven decentralized spectrum sharing in 5G networks：Potential game approach[J].IEEE Transactions on Vehicular Technology，2017，66（9）：7797-7808.
[12] LU H，HE X，DU M，et al.Edge QoE：Computation off-loading with deep reinforcement learning for internet of things[J].IEEE Internet of Things Journal，2020，7（10）：9255-9265.
[13] LI J，GAO H，LV T，et al.Deep reinforcement learning based computation offloading and resource allocation for MEC[C]//Proceedings of the 2018 IEEE Wireless Communications and Networking Conference，2018：1-6.
[14] ZHENG J，CAI Y，LIUY，et al.Optimal power allocation and user scheduling in multicell networks：Base station cooperation using a game-theoretic approach[J].IEEE Transactions on Wireless Communications，2014，13（12）：6928-6942.
[15] KSAIRI N，BIANCHI P，CIBLAT P，et al.Resource allocation for downlink cellular OFDMA systems-part I：Optimal allocation[J].IEEE Transactions on Singal Processing，2010，58（2）：735-749.
[16] LI Y，QI F，WANG Z，et al.Distributed edge computing offloading algorithm based on deep reinforcement lea-rning[J].IEEE Access，2020，8：85204-85215.
[17] GRONDMAN I，BUSONIU L，LOPES G，et al.A survey of actor-critic reinforcement learning：Standard and natural policy gradients[J].IEEE Transactions on Systems Man & Cybernetics（Part C），2012，42（6）：1291-1307.
[18] WANG Y，FANG W，DING Y，et al.Computation offloa-ding optimization for UAV-assisted mobile edge com-puting：A deep deterministic policy gradient approach[J].Wireless Networks，2021，27（4）：1-16.
[19] TRAN T X，POMPILID.Joint task offloading and resource allocation for multi-server mobile-edge computing networks[J].IEEE Transactions on Vehicular Technology，2017，68（1）：856-868.
[20] YIN W，SCHUTZE H，XIANG B，et al.ABCNN：Attention-based convolutional neural network for modeling sentence pairs[J].Transactions of the Association for Computational Linguistics，2016，4：259-272.
[21] CHEN X，GE H，LIU L，et al.Computing offloading decision based on DDPG algorithm in mobile edge computing[C]//Proceedings of 2021 IEEE 6th International Conference on Cloud Computing and Big Data Analytics，2021：391-399.
[22] LIU Y，YU H，XIE S，et al.Deep reinforcement learning for offloading and resource allocation in vehicle edge computing and networks[J].IEEE Transactions on Vehicular Technology，2019，68（11）：11158-11168.