Abstract

We present a dynamic programming approach for the solution of first-order Markov decisions processes. This technique uses an MDP whose dynamics is represented in a variant of the situation calculus allowing for stochastic actions. It produces a logical description of the optimal value function and policy by constructing a set of first-order formulae that minimally partition state space according to distinctions made by the value function and policy. This is achieved through the use of an operation known as decision-theoretic regression. In effect, our algorithm performs value iteration without explicit enumeration of either the state or action spaces of the MDP. This allows problems involving relational fluents and quantification to be solved without requiring explicit state space enumeration or conversion to propositional form. 1

Citations

1964	Dynamic Programming - Bellman - 1957
878	Markov Decision Processes: Discrete Stochastic Dynamic Programming - Puterman - 1994
519	The frame problem in the situation calculus: A simple solution (sometimes) and a completeness result for goal regression - Reiter - 1991
182	Situations and actions and causal laws - McCarthy - 1963
152	KNOWLEDGE IN ACTION: Logical Foundations for Describing and Implementing Dynamical Systems - Reiter
146	SPUDD: Stochastic planning using decision diagrams - Hoey, St-Aubin, et al. - 1999
133	Decision theoretic planning: Structural assumptions and computational leverage - Boutilier, Dean, et al. - 1999
120	Stochastic dynamic programming with factored representations - Boutilier, Dearden, et al.
119	The Independent Choice Logic for modelling multiple agents under uncertainty - Poole - 1997
88	Decision-theoretic, high-level agent programming in the situation calculus - Boutilier, Reiter, et al. - 2000
81	Some contributions to the metatheory of the situation calculus - Pirri, Reiter - 1999
51	Reasoning about noisy sensors in the situation calculus - Bacchus, Halpern, et al. - 1995
34	High-level planning and control with incomplete information using POMDPs - H, Bonet - 1998