Reinforcement Learning in POMDP's via Direct Gradient Ascent (2000)
| Venue: | In Proc. 17th International Conf. on Machine Learning |
| Citations: | 61 - 2 self |
BibTeX
@INPROCEEDINGS{Baxter00reinforcementlearning,
author = {Jonathan Baxter and Peter L. Bartlett},
title = {Reinforcement Learning in POMDP's via Direct Gradient Ascent},
booktitle = {In Proc. 17th International Conf. on Machine Learning},
year = {2000},
pages = {41--48},
publisher = {Morgan Kaufmann}
}
Years of Citing Articles
Bookmark
 |
 |
 |
 |
 |
 |
OpenURL
Abstract
This paper discusses theoretical and experimental aspects of gradient-based approaches to the direct optimization of policy performance in controlled POMDPs. We introduce GPOMDP, a REINFORCE-like algorithm for estimating an approximation to the gradient of the average reward as a function of the parameters of a stochastic policy. The algorithm's chief advantages are that it requires only a single sample path of the underlying Markov chain, it uses only one free parameter 2 [0; 1), which has a natural interpretation in terms of bias-variance trade-off, and it requires no knowledge of the underlying state. We prove convergence of GPOMDP and show how the gradient estimates produced by GPOMDP can be used in a conjugate-gradient procedure to find local optima of the average reward. 1. Introduction "Reinforcement learning" is used to describe the general problem of training an agent to choose its actions so as to increase its long-term average reward. The structure of th...
