Abstract: As a genetics-based machine learning technique, Zeroth-level Classifier System（ZCS） has shown promise in applying to multi-step problems. However, the standard ZCS is based on a discounted reward reinforcement learning algorithm, which optimizes the discounted total reward received by an agent but is not suitable for all multi-step problems. There are some average reward reinforcement learning methods available, such as R-learning, which optimize the average reward per time step. In this paper, R-learning is used as the reinforcement learning employed by ZCS, to replace its discounted reward reinforcement learning approach. The modification results show classifier system can effectively prevent the occurrence of overgeneralization and support long action chains, and thus is able to solve large multi-step problems.

Key words: average reward, reinforcement learning, R-learning, Learning Classifier Systems（LCS）, Zeroth-level Classifier System（ZCS）, multi-step problems

摘要： 零阶学习分类元系统ZCS（Zeroth-level Classifier System）作为一种基于遗传的机器学习技术（Genetics-Based Machine Learning），在解决多步学习问题上，已展现出应用价值。然而标准的ZCS系统采用折扣奖赏强化学习技术，难于适应更为广泛的应用领域。基于ZCS的现有框架，提出了一种采用平均奖赏强化学习技术（R-学习算法）的分类元系统，将ZCS中的折扣奖赏强化学习方法替换为R-学习算法，从而使ZCS一方面可应用于需要优化平均奖赏的问题领域，另一方面则可求解规模较大、需要动作长链支持的多步学习问题。实验显示，在多步学习问题中，该系统可给出满意解，且在维持动作长链，以及克服过泛化问题方面，具有更优的特性。

关键词: 平均奖赏, 强化学习, R-学习算法, 学习分类元系统（LCS）, 零阶分类元系统（ZCS）, 多步学习问题