Reinforcement learning can provide a robust and natural means for agents to learn how to coordinate their action choices in multiagent systems. We examine some of the factors that can influence the dynamics of the learning process in such a setting. We first distinguish reinforcement learners that are unaware of (or ignore) the presence of other agents from those that explicitly attempt to learn the value of joint actions and the strategies of their counterparts. We study (a simple form of) Q-learning in cooperative multiagent systems under these two perspectives, focusing on the influence of that game structure and exploration strategies on convergence to (optimal and suboptimal) Nash equilibria. We then propose alternative optimistic exploration strategies that increase the likelihood of convergence to an optimal equilibrium. 1

: We examine the dynamic formation and stochastic evolution of networks connecting individuals. The payoff to an individual from an economic or social activity depends on the network of connections among individuals. Over time individuals form and sever links connecting themselves to other individuals based on the improvement that the resulting network offers them relative to the current network. We call such sequences of networks, improving paths,' and show that such sequences can include cycles and study conditions on underlying allocation rules that characterize cycles. Building on the concept of improving paths, we consider a stochastic evolutionary process where in addition to intended changes in the network there is a small probability of unintended changes or errors. Predictions can be made regarding the relative likelihood that the stochastic process will lead to any given network at some time, and the evolutionary process selects from among the statically stable networks and c...

I survey the recent literature on the formation of networks. I provide definitions of network games, a number of examples of models from the literature, and discuss some of what is known about the (in)compatibility of overall societal welfare with individual incentives to form and sever links.

Abstract not found

There has been a growing interest in AI in the design of multiagent systems, especially in multiagent cooperative planning. In this paper, we investigate the extent to which methods from single-agent planning and learning can be applied in multiagent settings. We survey a number of different techniques from decision-theoretic planning and reinforcement learning and describe a number of interesting issues that arise with regard to coordinating the policies of individual agents. To this end, we describe multiagent Markov decision processes as a general model in which to frame this discussion. These are special n-person cooperative games in which agents share the same utility function. We discuss coordination mechanisms based on imposed conventions (or social laws) as well as learning methods for coordination. Our focus is on the decomposition of sequential decision processes so that coordination can be learned (or imposed) locally, at the level of individual states. We also discuss the use of structured problem representations and their role in the generalization of learned conventions and in approximation. 1

Multiagent learning is a key problem in game theory and AI. It involves two interrelated learning problems: identifying the game and learning to play. These two problems prevail even in team games where the agents' interests do not conflict. Even team games can have multiple Nash equilibria, only some of which are optimal. We present optimal adaptive learning (OAL), the first algorithm that converges to an optimal Nash equilibrium for any team Markov game. We provide a convergence proof, and show that the algorithm's parameters are easy to set so that the convergence conditions are met. Our experiments show that existing algorithms do not converge in many of these problems while OAL does. We also demonstrate the importance of the fundamental ideas behind OAL: incomplete history sampling and biased action selection.

This paper presents on overview of the economics that lies behind social interaction models and briefly discusses the empirical approaches to social interactions. We present a simple model with local interactions, similar to Glaeser, Sacerdote and Scheinkman (1996) but using a continuous action space and starting with optimizing behavior. We then extend the model to include both global and local interactions. We suggest and use a methodology for using variation of intra-city aggregates to identify the relative sizes of local and global interactions. We also present a model with endogenous location choice and use the predictions of that model to identify the sources of cross-city variance that are due to sorting and interaction. Finally, we present a brief discussion of using time-series to estimate the social interactions in broad aggregates.

This article argues that the agent-based computational model permits a distinctive approach to social science for which the term “generative ” is suitable. In defending this terminology, features distinguishing the approach from both “inductive ” and “deductive ” science are given. Then, the following specific contributions to social science are discussed: The agent-based computational model is a new tool for empirical research. It offers a natural environment for the study of connectionist phenomena in social science. Agent-based modeling provides a powerful way to address certain enduring—and especially interdisciplinary—questions. It allows one to subject certain core theories—such as neoclassical microeconomics—to important types of stress (e.g., the effect of evolving preferences). It permits one to study how rules of individual behavior give rise—or “map up”—to macroscopic regularities and organizations. In turn, one can employ laboratory behavioral research findings to select among competing agent-based (“bottom up”) models. The agent-based approach may well have the important effect of decoupling individual rationality from macroscopic equilibrium and of separating decision science from social science more generally. Agent-based modeling offers powerful new forms of hybrid theoretical-computational work; these are particularly relevant to the study of non-equilibrium systems. The agentbased approach invites the interpretation of society as a distributed computational device, and in turn the interpretation of social dynamics as a type of computation. This interpretation raises important foundational issues in social science—some related to intractability, and some to undecidability proper. Finally, since “emergence” figures prominently in this literature, I take up the connection between agent-based modeling and classical emergentism, criticizing the latter and arguing that the two are incompatible. � 1999 John Wiley &

Abstract not found

This paper develops an approach to equilibrium selection in game theory based on studying the equilibriating process through which equilibrium is achieved. The differential equations derived from models of interactive learning typically have stationary states that are not isolated. Instead, Nash equilibria that specify the same behavior on the equilibrium path, but different out-of-equilibrium behavior, appear in connected components of stationary states. The stability properties of these components often depend critically on the perturbations to which the system is subjected. We argue that it is then important to incorporate such drift into the model. A su±cient condition is provided for drift to create stationary states with strong stability properties near a component of equilibria. This result is used to derive comparative static predictions concerning common questions raised in the literature on refinements of Nash equilibrium