Abstract

Exploration plays a fundamental role in any active learning system. This study evaluates the role of exploration in active learning and describes several local techniques for exploration in finite, discrete domains, embedded in a reinforcement learning framework (delayed reinforcement). This paper distinguishes between two families of exploration schemes: undirected and directed exploration. While the former family is closely related to random walk exploration, directed exploration techniques memorize exploration-specific knowledge which is used for guiding the exploration search. In many finite deterministic domains, any learning technique based on undirected exploration is inefficient in terms of learning time, i.e. learning time is expected to scale exponentially with the size of the state space (Whitehead, 1991b) . We prove that for all these domains, reinforcement learning using a directed technique can always be performed in polynomial time, demonstrating the important role of e...

Citations

1447	A New Approach to Linear Filtering and Prediction Problems - Kalman, Emil - 1960
1135	Learning from delayed rewards - Watkins - 1989
1060	Learning to predict by the methods of temporal differences - Sutton - 1988
434	Dynamic programming and Markov processes - Howard - 1960
427	Integrated architectures for learning, planning, and reacting based on approximating integrated architectures for learning, planning, and reacting based on approximating dynamic programming - Sutton - 1990
316	Automatic programming of behavior-based robots using reinforcement learning - Mahadevan, Connell - 1992
280	Learning in embedded systems - Kaelbling - 1993
255	Self-improving reactive agents based on reinforcement learning, planning, and teaching - Lin - 1992
219	Temporal Credit Assignment in Reinforcement Learning - Sutton - 1984
185	Learning and sequential decision making - Barto, Sutton - 1990
173	Reinforcement learning with perceptual aliasing: The perceptual distinctions approach - Chrisman - 1992
151	Transfer of learning by composing solutions of elemental sequential tasks - Singh - 1992
147	Learning to predict by the methods of temporal di erence - Sutton - 1988
117	input generalization in delayed reinforcement learning: an algorithm and performance comparisons - Chapman, Kaelbling
105	Realtime learning and control using asynchronous dynamic programmming - Baro, Bradtke, et al. - 1991
94	Efficient Memory-based Learning for Robot Control. Unpublished - Moore - 1990
85	The Truck Backer-Upper: An example of Self-Learning in Neural Networks - Nguyen, Widrow - 1989
80	Learning to perceive and act by trial and error - Whitehead, Ballard - 1991
64	Learning to control an unstable system with forward modeling - Jordan, Jacobs - 1990
63	Diversity-based inference of finite automata - Schapire - 1988
58	Active Exploration in Dynamic Environments - Thrun, Moeller - 1992
49	Learning and problem solving with multilayer connectionist systems - Anderson - 1986
39	Scaling reinforcement learning to robotics by exploiting the subsumption architecture - Mahadevan, Connell - 1991
33	A new approach to linear ltering and prediction problems, Trans - Kalman - 1960
32	Self-improving reactive agents:: Case studies of reinforcement learning frameworks - Lin - 1990
30	Discovering the structure of a reactive environment by exploration - Mozer, Bachrach - 1989
28	A study of cooperative mechanisms for faster reinforcement learning - Whitehead - 1991
27	A dual back-propagation scheme for scalar reward learning - Munro - 1987
26	Complexity and cooperation in Q-learning - Whitehead - 1901
24	On the computational economics of reinforcement learning - Barto, Singh
21	Self-improvement based on reinforcement learning, planning and teaching - Lin - 1991
21	Learning control of finite Markov chains with unknown transition probabilities - Sato, Abe, et al. - 1982
14	Adaptive confidence and adaptive curiosity - Schmidhuber - 1991
14	Neural network based process optimization and control - Sofge, White - 1990
13	Generic constraints on underspecified target trajectories - Jordan - 1989
13	Knowledge of knowledge and intelligent experimentation for learning control - Moore - 1991
11	Connectionist Modeling and Control of Finite State Environments - Bachrach - 1992
10	On planning and exploration in non-discrete environments - Thrun, Moller - 1991
10	Planning with an adaptive world model - Thrun, Moller, et al. - 1991
6	A neurally-inspired connectionist approach to learning and performance in vision-based robot motion planning - Murphy - 1989
6	Diversity-based inference of nite automata - Rivest, Schapire - 1994
4	Connectionist modeling and control of nite state systems given partial state information - Bachrach, Mozer - 1991
4	Generic constraints on underspeci ed target trajectories - Jordan - 1989
3	Adaptive con dence and adaptive curiosity - Schmidhuber - 1991
1	Markov Decision Process with Policy Constraints - Nafeh - 1976
1	Learning control of nite markov chains with explicit trade-o between estimation and control - Sato, Abe, et al. - 1990