Abstract

This paper presents a new approach to hierarchical reinforcement learning based on the MAXQ decomposition of the value function. The MAXQ decomposition has both a procedural semantics---as a subroutine hierarchy---and a declarative semantics---as a representation of the value function of a hierarchical policy. MAXQ unifies and extends previous work on hierarchical reinforcement learning by Singh, Kaelbling, and Dayan and Hinton. Conditions under which the MAXQ decomposition can represent the optimal value function are derived. The paper defines a hierarchical Q learning algorithm, proves its convergence, and shows experimentally that it can learn much faster than ordinary "flat" Q learning. Finally, the paper discusses some interesting issues that arise in hierarchical reinforcement learning including the hierarchical credit assignment problem and non-hierarchical execution of the MAXQ hierarchy. 1 Introduction Hierarchical approaches to reinforcement learning (RL) problems promise ma...

Citations

342	Dynamic Programming and Optimal Control, Athena Scientific - Bertsekas - 2000
222	Motivated reinforcement learning - Dayan - 2002
212	Reinforcement learning with hierarchies of machines - Parr, Russell - 1998
186	On the convergence of stochastic iterative dynamic programming algorithms - Jaakkola, Jordan, et al. - 1994
151	Transfer of learning by composing solutions of elemental sequential tasks - Singh - 1992
103	Decomposition techniques for planning in stochastic domains - Dean, Lin - 1995
91	Convergence results for single-step on-policy reinforcement-learning algorithms - Singh - 2000
81	Incremental multi-step Q-learning - Peng, Williams - 1994
25	Hierarchical reinforcement learning: Preliminary results - Kaelbling - 1993