DMCA
Online Learning in Markov Decision Processes with Changing Cost Sequences
| Citations: | 1 - 0 self |
BibTeX
@MISC{Dick_onlinelearning,
author = {Travis Dick and András György and Csaba Szepesvári},
title = {Online Learning in Markov Decision Processes with Changing Cost Sequences},
year = {}
}
Share
 |
 |
 |
 |
||
OpenURL
Abstract
In this paper we consider online learning in finite Markov decision processes (MDPs) with changing cost sequences under full and banditinformation. We propose to view this problem as an instance of online linear optimization. We propose two methods for this problem: MD 2 (mirror descent with approximate projections) and the continuous exponential weights algorithm with Dikin walks. We provide a rigorous complexity analysis of these techniques, while providing near-optimal regret-bounds (in particular, we take into account the computational costs of performing approximate projections in MD 2). In the case of full-information feedback, our results complement existing ones. In the case of bandit-information feedback we consider the online stochastic shortest path problem, a special case of the above MDP problems, and manage to improve the existing results by removing the previous restrictive assumption that the statevisitation probabilities are uniformly bounded away from zero under all policies. 1.
Keyphrases
cost sequence online learning markov decision approximate projection online linear optimization rigorous complexity analysis mdp problem continuous exponential weight algorithm bandit-information feedback path problem full-information feedback previous restrictive assumption near-optimal regret-bounds dikin walk computational cost statevisitation probability special case mirror descent finite markov decision process
