Abstract not found

Due to the increasing sophistication and miniaturization of computational components, complex, distributed systems of interacting agents are becoming ubiquitous. Such systems, where each agent aims to optimize its own performance, but where there is a welldefined set of system-level performance criteria, are called collectives. The fundamental problem in analyzing/designing such systems is in determining how the combined actions of a large number of agents leads to “coordinated ” behavior on the global scale. Examples of artificial systems which exhibit such behavior include packet routing across a data network, control of an array of communication satellites, coordination of multiple rovers, and dynamic job scheduling across a distributed computer grid. Examples of natural systems include ecosystems, economies, and the organelles within a living cell. No current scientific discipline provides a thorough understanding of the relation between the structure of collectives and how well they meet their overall performance criteria. Although still very young, research on collectives has resulted in successes both in understanding and designing such systems. It is expected that as it matures and draws upon other disciplines related to collectives, this field will greatly expand the range of computationally addressable tasks. Moreover, in addition to drawing on them, such a fully developed field of collective intelligence may provide insight into already established scientific fields, such as mechanism design, economics, game theory, and population biology. This chapter provides a survey to the emerging science of collectives.

We consider the problem of designing the the utility functions of the utility-maximizing agents in a multi-agent system (MAS) so that they work synergistically to maximize a global utility. The particular problem domain we explore is the control of network routing by placing agents on all the routers in the network. Conventional approaches to this task have the agents all use the Ideal Shortest Path routing Algorithm (ISPA). We demonstrate that in many cases, due to the side-effects of one agent’s actions on another agent’s performance, having agents use ISPA’s is suboptimal as far as global aggregate cost is concerned, even when they are only used to route in£nitesimally small amounts of traf£c. The utility functions of the individual agents are not “aligned” with the global utility, intuitively speaking. As a particular example of this we present an instance of Braess’ paradox in which adding new links to a network whose agents all use the ISPA results in a decrease in overall throughput. We also demonstrate that load-balancing, in which the agents ’ decisions are collectively made to optimize the global cost incurred by all traf£c currently being routed, is suboptimal as far as global cost averaged across time is concerned. This is also due to “side-effects”, in this case of current routing decision on future traf£c. The mathematics of Collective

Single-agent reinforcement learners in time-extended domains and multi-agent systems share a common difficulty known as the credit assignment problem. Multiagent systems have the structural credit assignment problem of determining the contributions of a particular agent to a common task. Instead, time-extended single-agent systems have the temporal credit assignment problem of determining the contribution of a particular action to the quality of the full sequence of actions. Traditionally these two problems are considered different and are handled in separate ways. In this article we show how these two forms of the credit assignment problem are equivalent. In this unified framework, a single-agent Markov decision process can be broken down into a single-time-step multiagent process. Furthermore we show that Monte Carlo estimation or Q-learning (depending on whether the values of resulting actions in the episode are known at the time of learning) are equivalent to different agent utility functions in a multi-agent system. This equivalence shows how an often neglected issue in multi-agent systems is equivalent to a well-known deficiency in multi-timestep learning and lays the basis for solving time-extended multi-agent problems, where both credit assignment problems are present. 1.

Dedicated to my parents and family.

Group utility functions are an extension of the common team utility function for providing multiple agents with a common reinforcement learning signal for learning cooperative behaviour. In this paper we describe what group utility functions are and suggest using them to provide non-player computer game character behaviours. As yet, reinforcement learning techniques have very rarely been used for computer game character specification. Here we show the results of using a group utility function to learn an equilibrium between two computer game characters and compare this against the performance of the two agents learning independently. We also explain how group utility functions could be applied to learn equilibria between groups of agents. We highlight some implementation issues arising from using a commercial computer game engine for multi-agent reinforcement learning experiments. 1