DMCA
Learning and Sequential Decision Making (1989)
| Venue: | LEARNING AND COMPUTATIONAL NEUROSCIENCE |
| Citations: | 205 - 11 self |
BibTeX
@INPROCEEDINGS{Barto89learningand,
author = {Andrew G. Barto and R. S. Sutton and C. J. C. H. Watkins},
title = {Learning and Sequential Decision Making},
booktitle = {LEARNING AND COMPUTATIONAL NEUROSCIENCE},
year = {1989},
pages = {539--602},
publisher = {MIT Press}
}
Share
 |
 |
 |
 |
||
OpenURL
Abstract
In this report we show how the class of adaptive prediction methods that Sutton called "temporal difference," or TD, methods are related to the theory of squential decision making. TD methods have been used as "adaptive critics" in connectionist learning systems, and have been proposed as models of animal learning in classical conditioning experiments. Here we relate TD methods to decision tasks formulated in terms of a stochastic dynamical system whose behavior unfolds over time under the influence of a decision maker's actions. Strategies are sought for selecting actions so as to maximize a measure of long-term payoff gain. Mathematically, tasks such as this can be formulated as Markovian decision problems, and numerous methods have been proposed for learning how to solve such problems. We show how a TD method can be understood as a novel synthesis of concepts from the theory of stochastic dynamic programming, which comprises the standard method for solving such tasks when a model of the dynamical system is available, and the theory of parameter estimation, which provides the appropriate context for studying learning rules in the form of equations for updating associative strengths in behavioral models, or connection weights in connectionist networks. Because this report is oriented primarily toward the non-engineer interested in animal learning, it presents tutorials on stochastic sequential decision tasks, stochastic dynamic programming, and parameter estimation.
Keyphrases
sequential decision making td method parameter estimation animal learning stochastic dynamic programming connectionist network associative strength stochastic sequential decision task markovian decision problem connection weight numerous method dynamical system stochastic dynamical system behavioral model temporal difference behavior unfolds decision maker learning rule standard method adaptive prediction method adaptive critic appropriate context long-term payoff gain classical conditioning experiment squential decision making novel synthesis decision task
