Abstract

This paper presents the HDG learning algorithm, which uses a hierarchical decomposition of the state space to make learning to achieve goals more efficient with a small penalty in path quality. Special care must be taken when performing hierarchical planning and learning in stochastic domains, because macro-operators cannot be executed ballistically. The HDG algorithm, which is a descendent of Watkins' Q-learning algorithm, is described here and preliminary empirical results are presented. 1 INTRODUCTION Reinforcement learning is a general tool for deriving strategies that optimize a fixed reinforcement function in a stochastic environment. A crucial problem in reinforcement learning is temporal credit assignment: how to choose actions based on good results that happen after (perhaps long after) the action is taken. This problem is solved well in the general case by temporal difference methods, such as Watkins' Q learning [Barto et al., 1989, Watkins, 1989] and Sutton's TD ...

Citations

7321	Introduction to Algorithms - Cormen, Leiserson, et al. - 2001
1611	Computational geometry: An introduction - Preparata, Shamos - 1985
1135	Learning from delayed rewards - Watkins - 1989
1060	Learning to predict by the methods of temporal differences - Sutton - 1988
427	Integrated architectures for learning, planning, and reacting based on approximating integrated architectures for learning, planning, and reacting based on approximating dynamic programming - Sutton - 1990
222	Motivated reinforcement learning - Dayan - 2002
185	Learning and sequential decision making - Barto, Sutton - 1990
178	Reinforcement Learning for Robots Using Neural Networks - Lin - 1993
125	The role of exploration in learning control - Thrun - 1992
14	A role for anticipation in reactive systems that learn - Whitehead, Ballard - 1989
1	Pack 1993a. Learning in Embedded Systems - Kaelbling - 1990
1	Pack 1993b. Learning to achieve goals - Kaelbling
1	Parallel and dynamic approximation schemes for planar shortest paths - Klein, Sairam - 1992
1	Pal 1992a. Reinforcement learning with a hierarchy of abstract models - Singh
1	Pal 1992b. Scaling reinforcement learning algorithms by learning variable temporal resolution models - Singh
1	Pal 1992c. Transfer of learning by composing solutions of elemental sequential tasks. Machine Learning 8(3):323--340 - Singh