DMCA
Dual Representations for Dynamic Programming
| Citations: | 2 - 0 self |
BibTeX
@MISC{Wang_dualrepresentations,
author = {Tao Wang and Daniel Lizotte and Michael Bowling and Dale Schuurmans},
title = {Dual Representations for Dynamic Programming},
year = {}
}
Share
 |
 |
 |
 |
||
OpenURL
Abstract
We propose a dual approach to dynamic programming and reinforcement learning, based on maintaining an explicit representation of visit distributions as opposed to value functions. An advantage of working in the dual is that it allows one to exploit techniques for representing, approximating, and estimating probability distributions, while also avoiding any risk of divergence. We begin by formulating a modified dual of the standard linear program that guarantees the solution is a globally normalized visit distribution. Using this alternative representation, we then derive dual forms of dynamic programming, including on-policy updating, policy improvement and off-policy updating, and furthermore show how to incorporate function approximation. We then investigate the convergence properties of these algorithms, both theoretically and empirically, and show that the dual approach remains stable in situations when primal value function approximation diverges. Overall, the dual approach offers a viable alternative to standard dynamic programming techniques and offers new avenues for developing algorithms for sequential decision making.
Keyphrases
dynamic programming dual approach dual representation function approximation explicit representation on-policy updating reinforcement learning dynamic programming technique probability distribution dual form value function modified dual sequential decision making new avenue policy improvement visit distribution convergence property viable alternative primal value function approximation diverges normalized visit distribution alternative representation off-policy updating standard linear program
