Abstract

This dissertation is submitted for consideration for the dwree of Doctor' of Philosophy at the Uziver'sity of Cambr'idge Summary This thesis is concerned with practical issues surrounding the application of reinforcement lear'ning techniques to tasks that take place in high dimensional continuous state-space environments. In particular, the extension of on-line updating methods is considered, where the term implies systems that learn as each experience arrives, rather than storing the experiences for use in a separate off-line learning phase. Firstly, the use of alternative update rules in place of standard Q-learning (Watkins 1989) is examined to provide faster convergence rates. Secondly, the use of multi-layer perceptton (MLP) neural networks (Rumelhart, Hinton and Williams 1986) is investigated to provide suitable generalising function approximators. Finally, consideration is given to the combination of Adaptive Heuristic Critic (AHC) methods and Q-learning to produce systems combining the benefits of real-valued actions and discrete switching

Citations

2441	A robust layered control system for a mobile robot - Brooks - 1986
1965	Dynamic Programming - Bellman - 1957
1060	Learning to predict by the methods of temporal differences - Sutton - 1988
976	Multilayer feedforward networks are universal approximators - Hornik, Stinchcombe, et al. - 1989
772	Real-time obstacle avoidance for manipulators and mobile robots - Khatib - 1986
632	Approximation by superposition’s of a sigmoidal function - Cybenko
530	Pengi: An implementation of a theory of activity - Agre, Chapman - 1987
472	Learning to act using real-time dynamic programming - Barto, Bradtke, et al. - 1995
423	Neuron-like adaptive elements that can solve difficult learning problems - Barto, Sutton, et al. - 1983
368	A Learning Algorithm for Continually Running Fully Recurrent Neural Networks’, Neural computation - Williams, Zipser - 1989
334	Practical issues in temporal difference learning - Tesauro - 1992
331	Dynamic Programming: Deterministic and Stochastic Models - Bertsekas - 1987
307	Robot motion planning: a distributed representation approach - Barraquand, Latombe - 1991
306	Universal plans for reactive robots in unpredictable environments - Schoppers - 1987
285	On the approximate realization of continuous mappings by neural networks - Funahashi - 1989
280	Learning in embedded systems - Kaelbling - 1993
258	Escaping brittleness: The possibilities of general-purpose learning algorithms applied to parallel rule-based systems - Holland - 1986
255	Self-improving reactive agents based on reinforcement learning, planning, and teaching - Lin - 1992
224	On-line Q-learning using connectionist systems - Rummery, Niranjan - 1994
219	Temporal Credit Assignment in Reinforcement Learning - Sutton - 1984
215	Backpropagation through time: what it does and how to do it - Werbos - 2002
186	On the convergence of stochastic iterative dynamic programming algorithms - Jaakkola, Jordan, et al. - 1994
178	Reinforcement Learning for Robots Using Neural Networks - Lin - 1993
169	Pruning algorithms ± a survey - Reed - 1993
167	Task decomposition through competition in a modular architecture: The what and where in vision tasks - Jacobs, Jordan, et al. - 1990
166	Reinforcement learning with replacing eligibility traces - Singh, Sutton - 1996
151	Transfer of learning by composing solutions of elemental sequential tasks - Singh - 1992
143	Learning Automata: An Introduction - Narendra, Thathachar - 1989
142	A resource-allocating network for function interpolation - Platt - 1991
133	Bayesian Methods for Adaptive Models - MacKay - 1992
113	Efficient exploration in reinforcement learning - Thrun - 1992
93	ZCS: A Zeroth Level Classifier System - Wilson - 1994
85	Efficient learning and planning within the dyna framework - Peng, Williams - 1993
81	Incremental multi-step Q-learning - Peng, Williams - 1994
74	Advanced supervised learning in multi-layer perceptrons – From backpropagation to adaptive learning algorithms - Riedmiller - 1994
64	Learning to control an unstable system with forward modeling - Jordan, Jacobs - 1990
61	Hierarchies of adaptive experts - Jordan, Jacobs - 1992
58	The convergence of TD() for general - Dayan - 1992
58	Active Exploration in Dynamic Environments - Thrun, Moeller - 1992
57	Issues in using function approximation for reinforcement learning - Thrun, Schwartz - 1993
52	Reinforcement learning applied to linear quadratic regulation - Bradtke - 1993
48	Scaling up reinforcement learning for robot control - Lin - 1993
40	Modified policy iteration algorithms for discounted Markov decision problems - Puterman, Shin - 1978
39	Optimal brain surgeon and general network pruning - Hassinbi, Stork, et al. - 1992
36	Hierarchical learning of robot skills by reinforcement - Lin - 1993
35	Acquiring robot skills via reinforcement learning - Gullapalli, Franklin, et al. - 1994
35	A reinforcement connectionist approach to robot path finding in nonmaze-like environments - Millán, Torras - 1992
31	yIodular neural networks for learning context-dependent game strategies - Boyan - 1992
25	Q-learning with hidden-unit restarting - Anderson - 1993
23	Truncating temporal differences: On the efficient implementation of TD(λ) for reinforcement learning - Cichosz - 1995