关键词: 极限环, 强化学习, 倒立摆

Abstract: An important application of reinforcement learning in control systems is inverted pendulum.This paper points out that the common reinforcement learning algorithm will get into the limit cycles in the inverted pendulum system，which makes the algorithm incorrectly converge and destroy the stabilization of the optimal control policy.But the limit cycles problem is often ignored in many literatures as the goal of their algorithms is only to keep the pendulum stand in a given time.To overcome the limit cycles problem，this paper proposes a new reinforcement learning algorithm based on action continuity criterion.The algorithm revises the reinforcement signal and improves the exploration policy to overcome the negative effect of limit cycles in the inverted pendulum system.Simulation and actual control results of the double inverted pendulum system show the algorithm can not only control inverted pendulum successfully，but also keep the control policy stable.

Key words: limit cycles, reinforcement learning, inverted pendulum