Abstract

TD() is a popular family of algorithms for approximate policy evaluation in large MDPs. TD() works by incrementally updating the value function after each observed transition. It has two major drawbacks: it makes inefficient use of data, and it requires the user to manually tune a stepsize schedule for good performance. For the case of linear value function approximations and = 0, the Least-Squares TD (LSTD) algorithm of Bradtke and Barto (Bradtke and Barto, 1996) eliminates all stepsize parameters and improves data efficiency. This paper extends Bradtke and Barto's work in three significant ways. First, it presents a simpler derivation of the LSTD algorithm. Second, it generalizes from = 0 to arbitrary values of ; at the extreme of = 1, the resulting algorithm is shown to be a practical formulation of supervised linear regression. Third, it presents a novel, intuitive interpretation of LSTD as a model-based reinforcement learning technique.

Citations

2827	Reinforcement Learning: An Introduction - Sutton, Barto - 1998
1466	Numerical Recipes in C. The Art of Scientific Computing - PRESS, FLANNERY, et al. - 1988
1060	Learning to predict by the methods of temporal differences - Sutton - 1988
427	Integrated architectures for learning, planning, and reacting based on approximating integrated architectures for learning, planning, and reacting based on approximating dynamic programming - Sutton - 1990
275	Prioritized sweeping: Reinforcement learning with less data and less time - Moore, Atkeson - 1993
184	An analysis of temporal-difference learning with function approximation - Tsitsiklis, Roy - 1997
178	Reinforcement Learning for Robots Using Neural Networks - Lin - 1993
139	Linear least-squares algorithms for temporal difference learning - Bradtke, Barto - 1996
111	Reinforcement learning for dynamic channel allocation in cellular telephone systems - Singh, Bertsekas - 1997
56	A.W.: Learning evaluation functions for global optimization and boolean satisfability - Boyan, Moore - 1998
36	Gain adaptation beats least squares - Sutton - 1992
30	A comparison of direct and model-based reinforcement learning - Atkeson, Santamaria - 1997
27	Learning Evaluation Functions for Global Optimization - Boyan - 1998