Policy gradient algorithm based on internal structural MPOMDP model

doi:10.3778/j.issn.1002-8331.2009.07.007

Computer Engineering and Applications ›› 2009, Vol. 45 ›› Issue (7): 20-23.DOI: 10.3778/j.issn.1002-8331.2009.07.007

• 博士论坛 • Previous Articles Next Articles

Policy gradient algorithm based on internal structural MPOMDP model

ZHANG Run-mei^1,2,WANG Hao¹,ZHANG You-sheng¹,YAO Hong-liang¹,FANG Chang-sheng¹

1.Department of Computer Science and Technology，Hefei University of Technology，Hefei 230009，China
2.School of Electronics and Information Engineering，Anhui University of Architecture，Hefei 230022，China

Received:2008-10-22 Revised:2008-12-04 Online:2009-03-01 Published:2009-03-01
Contact: ZHANG Run-mei

基于内部结构MPOMDP模型的策略梯度学习算法

张润梅^1,2,王浩¹,张佑生¹,姚宏亮¹,方长胜¹

1.合肥工业大学计算机与科学技术系，合肥 230009
2.安徽建筑工业学院电子与信息工程学院，合肥 230022

通讯作者: 张润梅

Abstract

Abstract: For the improvement of knowledge representation ability and reasoning efficiency of MPOMDP model，a new kind of MPOMDP model is proposed based on internal structure of Agent.The internal structure and its evolvement of Agent are presented to improve the reasoning efficiency of the model by means the joint probability distribution of system as the local factorization forms of internal variables set.A MIS-GPOMDP algorithm is given by expanding GPI-POMDP to internal structural MPOMDP model to solve the internal structural MPOMDP model.The results of the experiment show that the high efficiency of the reasoning and convergence are found in MIS-GPOMDP algorithm.

Key words: arkov Decision Processes（MDP）, reinforcement learning, MPOMDP model, policy gradient algorithm

摘要： 为了提高MPOMDP模型的知识表示能力和推理效率，提出一种基于Agent内部结构的MPOMDP模型。该模型能表示Agent的内部结构及其时间演化，并通过将系统联合概率分布表示成每个Agent内部变量集的局部因式形式，以提高模型的推理效率。将GPI-POMDP算法扩展到基于内部结构的MPOMDP模型中，给出基于内部状态的多Agent策略梯度算法（MIS-GPOMDP），来求解基于内部结构的MPOMDP。实验结果表明MIS-GPOMDP算法具有较高的推理效率，且算法是收敛的。

关键词: 马尔可夫决策过程, 强化学习, MPOMDP模型, 策略梯度算法

ZHANG Run-mei^1,2,WANG Hao¹,ZHANG You-sheng¹,YAO Hong-liang¹,FANG Chang-sheng¹. Policy gradient algorithm based on internal structural MPOMDP model[J]. Computer Engineering and Applications, 2009, 45(7): 20-23.

张润梅^1,2,王浩¹,张佑生¹,姚宏亮¹,方长胜¹. 基于内部结构MPOMDP模型的策略梯度学习算法[J]. 计算机工程与应用, 2009, 45(7): 20-23.

[1]	WANG Xiao, TANG Lun, HE Xiaoyu, CHEN Qianbin. Multi-dimensional Resource Optimization of Service Function Chain Based on Deep Reinforcement Learning [J]. Computer Engineering and Applications, 2021, 57(4): 68-76.
[2]	LAI Jun, WEI Jingyi, CHEN Xiliang. Overview of Hierarchical Reinforcement Learning [J]. Computer Engineering and Applications, 2021, 57(3): 72-79.
[3]	MA Zhihao, ZHU Xiangbin. Research on Quasi-hyperbolic Momentum Gradient for Adversarial Deep Reinforcement Learning [J]. Computer Engineering and Applications, 2021, 57(24): 90-99.
[4]	LI Baoshuai, YE Chunming. Job Shop Scheduling Problem Based on Deep Reinforcement Learning [J]. Computer Engineering and Applications, 2021, 57(23): 248-254.
[5]	WANG Jun, CAO Lei, CHEN Xiliang, LAI Jun, ZHANG Legui. Overview on Reinforcement Learning of Multi-agent Game [J]. Computer Engineering and Applications, 2021, 57(21): 1-13.
[6]	CHENG Yi, HAO Mimi. Path Planning for Indoor Mobile Robot with Improved Deep Reinforcement Learning [J]. Computer Engineering and Applications, 2021, 57(21): 256-262.
[7]	KUANG Liqun, LI Siyuan, FENG Li, HAN Xie, XU Qingyu. Application of Deep Reinforcement Learning Algorithm on Intelligent Military Decision System [J]. Computer Engineering and Applications, 2021, 57(20): 271-278.
[8]	KONG Songtao, LIU Chichi, SHI Yong, XIE Yi, WANG Kun. Review of Application Prospect of Deep Reinforcement Learning in Intelligent Manufacturing [J]. Computer Engineering and Applications, 2021, 57(2): 49-59.
[9]	LI Hao, NING Haoyu, KANG Yan, LIANG Wentao, HUO Wen. SMRFGAN Model for Text Emotion Transfer [J]. Computer Engineering and Applications, 2021, 57(2): 170-176.
[10]	ZHANG Rongxia, WU Changxu, SUN Tongchao, ZHAO Zengshun. Progress on Deep Reinforcement Learning in Path Planning [J]. Computer Engineering and Applications, 2021, 57(19): 44-56.
[11]	YANG Xueyu, CHEN Jianping, FU Qiming, LU You, WU Hongjie. Deep Deterministic Policy Gradient Algorithm Based on Stochastic Variance Reduction Method [J]. Computer Engineering and Applications, 2021, 57(19): 104-111.
[12]	SONG Haonan, ZHAO Gang, WANG Xingfen. Knowledge Reasoning Method Combining Knowledge Representation with Deep Reinforcement Learning [J]. Computer Engineering and Applications, 2021, 57(19): 189-197.
[13]	WANG Keyin, SHI Zhen, YANG Zhengcai, YANG Yahui, WANG Sishan. Path Planning for Mobile Robot Using Improved Reinforcement Learning Algorithm [J]. Computer Engineering and Applications, 2021, 57(18): 270-274.
[14]	ZHANG Jun, ZHU Qingwei, YAN Junjie, WEN Bo. UAV Indoor 3D Track Planning Based on Improved Reinforcement Learning Algorithm [J]. Computer Engineering and Applications, 2021, 57(16): 175-181.
[15]	CHE Xiangbei, KANG Wenqian, OUYANG Yuhong, YANG Kehan, LI Jian. SDN Routing Optimization Algorithm Based on Reinforcement Learning [J]. Computer Engineering and Applications, 2021, 57(12): 93-98.

Policy gradient algorithm based on internal structural MPOMDP model

基于内部结构MPOMDP模型的策略梯度学习算法

PDF

Knowledge

Abstract

Cite this article

share this article

References

Related Articles 15

Recommended Articles

Metrics