Analytical Mean Squared Error Curves for Temporal Difference Learning

DMCA

Analytical Mean Squared Error Curves for Temporal Difference Learning (1988)

Cached

Download Links

Other Repositories/Bibliography

DBLP

by Satinder Singh , Peter Dayan , G. Barto

Venue:	Machine Learning
Citations:	26 - 3 self

BibTeX

@INPROCEEDINGS{Singh88analyticalmean,
    author = {Satinder Singh and Peter Dayan and G. Barto},
    title = {Analytical Mean Squared Error Curves for Temporal Difference Learning},
    booktitle = {Machine Learning},
    year = {1988},
    pages = {5--40}
}

OpenURL

Abstract

We provide analytical expressions governing changes to the bias and variance of the lookup table estimators provided by various Monte Carlo and temporal difference value estimation algorithms with offine updates over trials in absorbing Markov reward processes. We have used these expressions to develop software that serves as an analysis tool: given a complete description of a Markov reward process, it rapidly yields an exact mean-square-error curve, the curve one would get from averaging together sample mean-square-error curves from an infinite number of learning trials on the given problem. We use our analysis tool to illustrate classes of mean-square-error curve behavior in a variety of example reward processes, and we show that although the various temporal difference algorithms are quite sensitive to the choice of step-size and eligibility-trace parameters, there are values of these parameters that make them similarly competent, and generally good.

Keyphrases

temporal difference learning analytical mean squared error curve analysis tool various monte carlo various temporal difference algorithm analytical expression infinite number exact mean-square-error curve example reward markov reward process markov reward mean-square-error curve behavior complete description eligibility-trace parameter offine update temporal difference value estimation sample mean-square-error curve lookup table estimator

DMCA