Abstract

An issue that is critical for the application of Markov decision processes (MDPs) to realistic problems is how the complexity of planning scales with the size of the MDP. In stochastic environments with very large or even in-nite state spaces, traditional planning and reinforcement learning algorithms are often inapplicable, since their running time typically scales linearly with the state space size in the worst case. In this paper we present a new algorithm that, given only a generative model (simulator) for an arbitrary MDP, performs near-optimal planning with a running time that has no dependence on the number of states. Although the running time is exponential in the horizon time (which depends only on the discount factor and the desired degree of approximation to the optimal policy), our results establish for the rst time that there are no theoretical barriers to computing near-optimal policies in arbitrarily large, unstructured MDPs. 1

Citations

2959	Artificial Intelligence: a Modern Approach - Russell, Norvig - 1995
2172	The Design and Analysis of Computer Algorithms - Aho, Hopcroft, et al. - 1974
649	Reinforcement learning - Sutton, Barto - 1998
472	Learning to act using real-time dynamic programming - Barto, Bradtke, et al. - 1995
362	Real-time heuristic search - Korf - 1990
228	Tractable inference for complex stochastic processes - Boyen, Koller - 1998
200	Exploiting structure in policy construction - Boutilier, Dearden, et al. - 1995
127	A robust and fast action selection mechanism for planning - Bonet, Loerincs, et al. - 1997
97	Approximate planning in large POMDPs via reusable trajectories - Kearns, Mansour, et al. - 2000
79	Computing factored value functions for policies in structured MDPs - Koller, Parr - 1999
70	Solving very large weakly coupled Markov decision processes - Meuleau, Hauskrecht, et al. - 1998
59	An upper bound on the loss from approximate optimal-value functions - Singh, Yee - 1994
35	Approximate Planning for Factored POMDPs using Belief State Simpli cation - McAllester, Singh - 1999
34	Finite-sample convergence rates for Q-learning and indirect algorithms - Kearns, Singh - 1998
27	Solving robot navigation problems with initial pose uncertainty using real-time heuristic search - KOENIG, SIMMONS - 1998
25	Integrating planning and execution in stochastic domains - Dearden, Boutilier - 1994
10	Arti cial Intelligence -AModern Approach - Russell, Norvig - 1995
5	Applying online-search to reinforcement learning - Davies, Ng, et al. - 1998