Abstract

An open problem in reinforcement learning is discovering hierarchical structure. HEXQ, an algorithm which automatically attempts to decompose and solve a model-free factored MDP hierarchically is described. By searching for aliased Markov sub-space regions based on the state variables the algorithm uses temporal and state abstraction to construct a hierarchy of interlinked smaller MDPs.

Citations

7321	Introduction to Algorithms - Cormen, Leiserson, et al. - 2001
2827	Reinforcement Learning: An Introduction - Sutton, Barto - 1998
878	Markov Decision Processes. Discrete Stochastic Dynamic Programming - Puterman - 1994
347	An introduction to cybernetics - Ashby - 1956
342	Decision-Theoretic Planning: Structural Assumptions and Computational Leverage - Boutilier, Dean, et al. - 1999
308	Adaptive Control Processes: a Guided Tour - Bellman - 1961
307	Hierarchical Reinforcement Learning with the MAXQ Value Function Decomposition - Dietterich - 1999
112	Hierarchical solution of markov decision processes using macro-actions - Hauskrecht, Meuleau, et al. - 1998
103	Decomposition techniques for planning in stochastic domains - Dean, Lin - 1995
98	Hierarchical Control and Learning for Markov Decision Processes - Parr - 1998
94	Hierarchical learning in stochastic domains: Preliminary results - Kaelbling - 1993
29	An overview of MAXQ hierarchical reinforcement learning - Dietterich
8	Generating Hierarchical Structure in Reinforcement Learning from State Variables - Hengst - 2000