Abstract

There has been a growing interest in AI in the design of multiagent systems, especially in multiagent cooperative planning. In this paper, we investigate the extent to which methods from single-agent planning and learning can be applied in multiagent settings. We survey a number of different techniques from decision-theoretic planning and reinforcement learning and describe a number of interesting issues that arise with regard to coordinating the policies of individual agents. To this end, we describe multiagent Markov decision processes as a general model in which to frame this discussion. These are special n-person cooperative games in which agents share the same utility function. We discuss coordination mechanisms based on imposed conventions (or social laws) as well as learning methods for coordination. Our focus is on the decomposition of sequential decision processes so that coordination can be learned (or imposed) locally, at the level of individual states. We also discuss the use of structured problem representations and their role in the generalization of learned conventions and in approximation. 1

Citations

1964	Dynamic Programming - Bellman - 1957
1060	Learning to predict by the methods of temporal differences - Sutton - 1988
878	Markov Decision Processes: Discrete Stochastic Dynamic Programming - Puterman - 1994
585	The evolution of cooperation. Basic - Axelrod - 1984
472	Learning to act using real-time dynamic programming - Barto, Bradtke, et al. - 1995
434	Dynamic programming and Markov processes - Howard - 1960
427	Integrated architectures for learning, planning, and reacting based on approximating integrated architectures for learning, planning, and reacting based on approximating dynamic programming - Sutton - 1990
417	Markov games as a framework for multi-agent reinforcement learning - Littman - 1994
398	Game Theory: Analysis of Conflict - Myerson - 1997
396	A model for reasoning about persistence and causation - Dean, Kanazawa - 1990
288	Planning and Control - Dean, Wellman - 1991
285	The Optimal Control of Partially Observable Markov Processes - Sondik - 1971
278	Convention: A Philosophical Study - Lewis - 1969
249	A General Theory of Equilibrium Selection in Games - Selten, Harsanyi - 1988
243	Acting optimally in partially observable stochastic domains - Cassandra, Kaelbling, et al. - 1994
235	The optimal control of partially observable Markov processes over a finite horizon - Smallwood, Sondik - 1973
231	Probabilistic robot navigation in partially observable environments - Simmons, Koenig - 1995
223	Generalization in reinforcement learning: safely approximating the value function - Boyan, Moore - 1995
222	Motivated reinforcement learning - Dayan - 2002
203	The parti-game algorithm for variable resolution reinforcement learning in multidimensional state-spaces - Moore, Atkeson - 1995
200	Exploiting structure in policy construction - Boutilier, Dearden, et al. - 1995
198	The Evolution of Conventions - Young - 1993
192	Game Theory - Owen - 1995
170	Rational Learning Leads to Nash Equilibrium - Kalai, Lehrer - 1993
158	On the synthesis of useful social laws for artificial agent societies - Shoham, Tennenholtz - 1992
151	Reward functions for accelerated learning - Mataric - 1994
135	Learning to coordinate without sharing information - Sen, Sekaran - 1994
125	Planning with deadlines in stochastic domains - Dean, Kaelbling, et al. - 1993
117	input generalization in delayed reinforcement learning: an algorithm and performance comparisons - Chapman, Kaelbling
103	Decomposition techniques for planning in stochastic domains - Dean, Lin - 1995
99	Optimal control of Markov decision processes with incomplete state estimation - Astrom - 1965
97	Reinforcement learning with soft state aggregation - Singh, Jaakkola, et al. - 1995
94	Computing optimal policies for partially observable decision processes using compact representations - Boutilier, Poole - 1996
81	Planning under uncertainty: Structural assumptions and computational leverage - Boutilier, Dean, et al. - 1996
78	An Adaptive Communication Protocol for Cooperating Mobile Robots - Yanco, Stein - 1993
76	Variable resolution dynamic programming: Efficiently learning action maps in multivariate real-valued state-spaces - Moore - 1991
65	Using abstractions for decision-theoretic planning with time constraints - Boutilier, Dearden - 1994
58	The convergence of TD() for general - Dayan - 1992
49	Divide and conquer in multi-agent planning - Ephrati, Rosenschein - 1994
49	Kaelbling. Learning in Embedded Systems - Pack - 1990
48	Emergent conventions in multi-agent systems: initial experimental results and observations (preliminary report - Shoham, Tennenholtz - 1992
47	2005): Essential Equilibria - Govindan, Wilson
40	Modified policy iteration algorithms for discounted Markov decision problems - Puterman, Shin - 1978
39	Steady State Learning and Nash Equilibrium - FUDENBERG, LEVINE - 1993
35	Approximating value trees in structured dynamic programming - Boutilier, Dearden - 1996
34	Control strategies for a stochastic planner - Tash, Russell - 1994
29	Multiagent coordination with learning classifier systems - Sen, Sekaran - 1996
27	Adaptive aggregation for infinite horizon dynamic programming - Bertsekas, Castanon - 1989
25	Integrating planning and execution in stochastic domains - Dearden, Boutilier - 1994
21	Moisés Goldszmidt. Exploiting structure in policy construction - Boutilier, Dearden - 1995