Abstract

Reinforcement learning algorithms based on the principles of Dynamic Programming (DP) have enjoyed a great deal of recent attention both empirically and theoretically. These algorithms have been referred to generically as Incremental Dynamic Programming (IDP) algorithms. IDP algorithms are intended for use in situations where the information or computational resources needed by traditional dynamic programming algorithms are not available. IDP algorithms attempt to find a global solution to a DP problem by incrementally improving local constraint satisfaction properties as experience is gained through interaction with the environment. This class of algorithms is not new, going back at least as far as Samuel's adaptive checkers-playing programs,...

Citations

2801	Optimization by simulated annealing - Kirkpatrick, Gelatt, et al. - 1983
1447	A New Approach to Linear Filtering and Prediction Problems - Kalman, Emil - 1960
1135	Learning from delayed rewards - Watkins - 1989
1060	Learning to predict by the methods of temporal differences - Sutton - 1988
620	Finite Markov Chains - Kemeny, Snell - 1976
545	Some Studies in Machine Learning using the Game of Checkers - Samuel - 2000
503	Principles of Artificial Intelligence - Nilsson - 1982
477	Tsitsiklis. Parallel and Distributed Computation: Numerical Methods - Bertsekas, N - 1989
473	Intractability and Time-Dependent Planning - Dean - 1987
472	Learning to act using real-time dynamic programming - Barto, Bradtke, et al. - 1995
434	Dynamic programming and Markov processes - Howard - 1960
427	Integrated architectures for learning, planning, and reacting based on approximating integrated architectures for learning, planning, and reacting based on approximating dynamic programming - Sutton - 1990
420	A stochastic approximation method - Robbins, Monro - 1951
362	Real-time heuristic search - Korf - 1990
334	Practical issues in temporal difference learning - Tesauro - 1992
331	Dynamic Programming: Deterministic and Stochastic Models - Bertsekas - 1987
327	Depth-first iterative-deepening: An optimal admissible tree search - Korf - 1985
308	Theory and Practice of Recursive Identification - Ljung, Söderström - 1983
306	Universal plans for reactive robots in unpredictable environments - Schoppers - 1987
275	Prioritized sweeping: Reinforcement learning with less data and less time - Moore, Atkeson - 1993
255	Self-improving reactive agents based on reinforcement learning, planning, and teaching - Lin - 1992
223	Introduction to Artificial Intelligence - Charniak, McDermott - 1985
219	Temporal Credit Assignment in Reinforcement Learning - Sutton - 1984
209	New Results in Linear Filtering and Prediction Theory,” Trans - Kalman, Bucy - 1961
196	Introduction to Stochastic Dynamic Programming - Ross - 1983
194	Linear Systems Theory and Design - Chen - 1999
190	Td-gammon, a self-teaching backgammon program, achieves master-level play - Tesauro - 1994
186	On the convergence of stochastic iterative dynamic programming algorithms - Jaakkola, Jordan, et al. - 1994
178	Adaptive Filtering Prediction and Control - Goodwin, Sin - 1985
173	Reinforcement learning with perceptual aliasing: The perceptual distinctions approach - Chrisman - 1992
157	Neuronlike elements that can solve difficult learning control problems - Barto, Sutton, et al. - 1983
151	Transfer of learning by composing solutions of elemental sequential tasks - Singh - 1992
145	Introduction Dynamic Systems: Theory, Models & Applications - Luenberger - 1979
129	Asynchronous stochastic approximation and Q-learning - Tsitsiklis - 1994
117	input generalization in delayed reinforcement learning: an algorithm and performance comparisons - Chapman, Kaelbling
96	Overcoming incomplete perception with utile distinction memory - McCallum - 1993
85	Receding horizon control of nonlinear systems - Mayne, Michalska - 1990
80	World Modeling for the Dynamic Construction of Real-Time Control Plans - Musliner, Durfee, et al. - 1995
75	Generalization of backpropagation with application to a recurrent gas market model, neural networks - Werbos - 1988
72	Universal planning: An (almost) universally bad idea - Ginsberg - 1989
72	Tight performance bounds on greedy policies based on imperfect value functions - Williams, Baird - 1994
71	Active perception and reinforcement learning - Whitehead, Ballard
65	Strategy learning with multilayer connectionist representations - Anderson - 1987
64	Learning to control an unstable system with forward modeling - Jordan, Jacobs - 1990
59	Memory approaches to reinforcement learning in non-Markovian domains - Lin, Mitchell - 1999
59	An upper bound on the loss from approximate optimal-value functions - Singh, Yee - 1994
58	The convergence of TD() for general - Dayan - 1992
55	Approximate Dynamic Programming for Real-Time Control and Neural Modeling,” Handbood of Intelligent Control, edited by - Werbos - 1992
55	Recursive Estimation and Time-Series Analysis - Young - 1984
53	Planning by incremental dynamic programming - Sutton - 1991