Abstract: An important application of reinforcement learning in control systems is inverted pendulum.This paper points out that the common reinforcement learning algorithm will get into the limit cycles in the inverted pendulum system，which makes the algorithm incorrectly converge and destroy the stabilization of the optimal control policy.But the limit cycles problem is often ignored in many literatures as the goal of their algorithms is only to keep the pendulum stand in a given time.To overcome the limit cycles problem，this paper proposes a new reinforcement learning algorithm based on action continuity criterion.The algorithm revises the reinforcement signal and improves the exploration policy to overcome the negative effect of limit cycles in the inverted pendulum system.Simulation and actual control results of the double inverted pendulum system show the algorithm can not only control inverted pendulum successfully，but also keep the control policy stable.

Key words: limit cycles, reinforcement learning, inverted pendulum

摘要： 倒立摆系统是强化学习的一种重要的应用领域。首先分析指出在倒立摆系统中，常用的强化学习算法存在着极限环问题，算法无法正确收敛、控制策略不稳定。但是由于在简单的一级倒立摆系统中算法的控制策略不稳定的现象还不明显，因此极限环问题常常被忽视。针对强化学习算法中极限环问题，提出基于动作连续性准则的强化学习算法。算法采用修正强化信号和改进探索策略的方法克服极限环对倒立摆系统的影响。将提出的算法用于二级倒立摆的实际系统控制中，实验结果证明算法不仅能成功控制倒立摆，而且可以保持控制策略的稳定。

关键词: 极限环, 强化学习, 倒立摆