关键词: 部分可观测马尔可夫决策过程, 表示学习, 强化学习, 对比预测编码, 深度双Q网络

Abstract: In the model unknown partially observable Markov decision process（POMDP）, the agent cannot directly access the true state of environment, and the perceptual uncertainty poses challenges for learning the optimal policy. Thus, a double deep Q-network reinforcement learning algorithm based on the representation of the contrastive predictive coding is proposed. The belief states are modeled explicitly to obtain a compact and efficient history encoding for the policy optimization. To improve data efficiency, the belief replay buffer is introduced to reduce the memory usage by directly storing the belief transition pairs instead of the observation and action sequences. In addition, the phased training strategy is designed for decoupling the representation learning from the policy learning process to improve training stability. The POMDP navigation tasks based on the Gym-MiniGrid environment are designed. Experimental results show that the semantic information related to the state can be captured by the proposed algorithm, which facilitates to achieve stable and efficient policy learning in POMDP.

Key words: partially observable Markov decision process（POMDP）, representation learning, reinforcement learning, contrastive predictive coding, double deep Q-network