Abstract

Most algorithms for solving POMDPs iteratively improve a value function that implicitly represents a policy and are said to search in value function space. This paper presents an approach to solving POMDPs that represents a policy explicitly as a finite-state controller and iteratively improves the controller by search in policy space. Two related algorithms illustrate this approach. The first is a policy iteration algorithm that can outperform value iteration in solving infinitehorizon POMDPs. It provides the foundation for a new heuristic search algorithm that promises further speedup by focusing computational effort on regions of the problem space that are reachable, or likely to be reached, from a start state. 1 Introduction A partially observable Markov decision process (POMDP) provides an elegant mathematical model for planning and control problems for which there can be uncertainty about the effects of actions and about the current state. It is well-known that ...

Citations

629	Planning and acting in partially observable stochastic domains - Kaelbling, Littman, et al. - 1998
285	The Optimal Control of Partially Observable Markov Processes - Sondik - 1971
235	The optimal control of partially observable Markov processes over a finite horizon - Smallwood, Sondik - 1973
51	A heuristic variable grid solution method of POMDPs - Brafman - 1997
34	Incremental methods for computing bounds in partially observable Markov decision processes - Hauskrecht - 1997
32	Heuristic search in restricted memory - Chakrabarti, Ghose, et al.
29	Finite-Memory Control of Partially Observable Systems - Hansen - 1998
28	An improved policy iteration algorithm for partially observable MDPs - Hansen - 1997
27	Incremental pruning: A simple, fast, exact algorithm for partially observable Markov decision processes - Cassandra, Littman - 1997
23	Efficient dynamic-programming updates in partially observable Markov decision processes - Littman, Cassandra, et al. - 1996
13	Incremental markov-model planning - Washington - 1996
1	A Decision Tree Approach to Maintaining a Deteriorating Physical System - Larsen - 1989