Abstract

The close connection between reinforcement learning (RL) algorithms and dynamic programming algorithms has fueled research on RL within the machine learning community. Yet, despite increased theoretical understanding, RL algorithms remain applicable to simple tasks only. In this paper I use the abstract framework afforded by the connection to dynamic programming to discuss the scaling issues faced by RL researchers. I focus on learning agents that have to learn to solve multiple structured RL tasks in the same environment. I propose learning abstract environment models where the abstract actions represent "intentions" of achieving a particular state. Such models are variable temporal resolution models because in different parts of the state space the abstract actions span different number of time steps. The operational definitions of abstract actions can be learned incrementally using repeated experience at solving RL tasks. I prove that under certain conditions s...

Citations

1135	Learning from delayed rewards - Watkins - 1989
1060	Learning to predict by the methods of temporal differences - Sutton - 1988
545	Some Studies in Machine Learning using the Game of Checkers - Samuel - 2000
472	Learning to act using real-time dynamic programming - Barto, Bradtke, et al. - 1995
427	Integrated architectures for learning, planning, and reacting based on approximating integrated architectures for learning, planning, and reacting based on approximating dynamic programming - Sutton - 1990
334	Practical issues in temporal difference learning - Tesauro - 1992
331	Dynamic Programming: Deterministic and Stochastic Models - Bertsekas - 1987
280	Learning in embedded systems - Kaelbling - 1993
219	Temporal Credit Assignment in Reinforcement Learning - Sutton - 1984
196	Introduction to Stochastic Dynamic Programming - Ross - 1983
151	Transfer of learning by composing solutions of elemental sequential tasks - Singh - 1992
117	input generalization in delayed reinforcement learning: an algorithm and performance comparisons - Chapman, Kaelbling
94	A heuristic approach to the discovery of macro-operators - Iba - 1989
76	Variable resolution dynamic programming: Efficiently learning action maps in multivariate real-valued state-spaces - Moore - 1991
71	Active perception and reinforcement learning - Whitehead, Ballard
61	Reinforcement learning with a hierarchy of abstract models - Singh - 1992
58	Active Exploration in Dynamic Environments - Thrun, Moeller - 1992
54	Learning to solve problems by searching for macro-operators - Korf - 1985
54	Learning complex, extended sequences using the principle of history compression - Schmidhuber - 1992
51	Two kinds of training information for evaluation function learning - Utgoff, Clouse - 1991
49	Reinforcement Learning and its application to control - Gullapalli - 1992
43	Consistency of HDP applied to a simple reinforcement learning problem - Werbos - 1990
30	Sequential Machines and Automata Theory - Booth - 1967
27	Sequential decision problems and neural networks - Barto, Sutton, et al. - 1990
26	Complexity and cooperation in Q-learning - Whitehead - 1901
24	On the computational economics of reinforcement learning - Barto, Singh
23	The ecient learning of multiple task sequences - Singh - 1992
21	Self-improvement based on reinforcement learning, planning and teaching - Lin - 1991
3	Prediction and prescription in systems modeling - Simon - 1990