Abstract

Reinforcement learning (RL) algorithms have traditionally been thought of as trial and error learning methods that use actual control experience to incrementally improve a control policy. Sutton's DYNA architecture demonstrated that RL algorithms can work as well using simulated experience from an environment model, and that the resulting computation was similar to doing one-step lookahead planning. Inspired by the literature on hierarchical planning, I propose learning a hierarchy of models of the environment that abstract temporal detail as a means of improving the scalability of RL algorithms. I present H-DYNA (Hierarchical DYNA), an extension to Sutton's DYNA architecture that is able to learn such a hierarchy of abstract models. H-DYNA differs from hierarchical planners in two ways: first, the abstract models are learned using experience gained while...

Citations

1060	Learning to predict by the methods of temporal differences - Sutton - 1988
478	Tsitsiklis. Parallel and Distributed Computation: Numerical Methods - Bertsekas, N - 1989
473	An Analysis of Time-Dependent Planning - Dean, Boddy - 1988
427	Integrated architectures for learning, planning, and reacting based on approximating integrated architectures for learning, planning, and reacting based on approximating dynamic programming - Sutton - 1990
411	Planning in a hierarchy of abstraction spaces - Sacerdoti - 1974
334	Practical issues in temporal difference learning - Tesauro - 1992
306	Universal plans for reactive robots in unpredictable environments - Schoppers - 1987
280	Learning in embedded systems - Kaelbling - 1993
196	Introduction to Stochastic Dynamic Programming - Ross - 1983
151	Transfer of learning by composing solutions of elemental sequential tasks - Singh - 1992
117	input generalization in delayed reinforcement learning: an algorithm and performance comparisons - Chapman, Kaelbling
105	Realtime learning and control using asynchronous dynamic programmming - Baro, Bradtke, et al. - 1991
94	A heuristic approach to the discovery of macro-operators - Iba - 1989
54	Learning to solve problems by searching for macro-operators - Korf - 1985
53	Planning by incremental dynamic programming - Sutton - 1991
24	On the computational economics of reinforcement learning - Barto, Singh
23	Scaling reinforcement learning algorithms by learning variable temporal resolution models - Singh - 1992