Efficient RL Algorithm by Combing AC with Dual Piecewise Model Learning
2 School of Computer Science and Engineering, Changshu Institute of Technology, Changshu, 215500
3 Jiangsu Province Key Laboratory of Intelligent Building Energy Efficiency, Suzhou University of Science and Technology, Suzhou, Jiangsu, 215006
4 School of Computer Science and Technology, Soochow University, Suzhou, Jiangsu, 215006
5 Collaborative Innovation Center of Novel Software Technology and Industrialization, Nanjing, 210000
6 Key Laboratory of Symbolic Computation and Knowledge Engineering of Ministry of Education, Jilin University, Changchun, 130012
7 Key Laboratory of Symbolic Computation and Knowledge Engineering of Ministry of Education, Jilin University, Changchun, 130012
8 College of Electronic & Information Engineering, Suzhou University of Science and Technology, Jiangsu, Suzhou, 215006
* Author to whom correspondence should be addressed.
As classic methods for handling continuous action space problem for continuous action space problem in RL, the actor-critic (AC) algorithm and its variants still fail to be sample efficiency. Therefore, we propose a method based on learning two linear models for planning. The two linear models refers to state-based piecewise model and action-based piecewise model, which are determined by the divisions for the state and action space, respectively. Through division, the models are learned more accurately. To accelerate the convergence, the sample near the goal is saved and used to learn the model, the value and the policy to balance the distribution of the samples. On two classic RL benchmarks with continuous MDPs, the proposed method shows the ability of learning an optimal policy by combing both models, and it also outperforms the representative methods in terms of convergence rate and sample efficiency.