## A survey of collectives (2004)

Venue: | IN COLLECTIVES AND THE DESIGN OF COMPLEX SYSTEMS |

Citations: | 19 - 10 self |

### BibTeX

@INPROCEEDINGS{Tumer04asurvey,

author = {Kagan Tumer and David Wolpert},

title = {A survey of collectives},

booktitle = {IN COLLECTIVES AND THE DESIGN OF COMPLEX SYSTEMS},

year = {2004},

pages = {1--42},

publisher = {Springer}

}

### Years of Citing Articles

### OpenURL

### Abstract

Due to the increasing sophistication and miniaturization of computational components, complex, distributed systems of interacting agents are becoming ubiquitous. Such systems, where each agent aims to optimize its own performance, but where there is a welldefined set of system-level performance criteria, are called collectives. The fundamental problem in analyzing/designing such systems is in determining how the combined actions of a large number of agents leads to “coordinated ” behavior on the global scale. Examples of artificial systems which exhibit such behavior include packet routing across a data network, control of an array of communication satellites, coordination of multiple rovers, and dynamic job scheduling across a distributed computer grid. Examples of natural systems include ecosystems, economies, and the organelles within a living cell. No current scientific discipline provides a thorough understanding of the relation between the structure of collectives and how well they meet their overall performance criteria. Although still very young, research on collectives has resulted in successes both in understanding and designing such systems. It is expected that as it matures and draws upon other disciplines related to collectives, this field will greatly expand the range of computationally addressable tasks. Moreover, in addition to drawing on them, such a fully developed field of collective intelligence may provide insight into already established scientific fields, such as mechanism design, economics, game theory, and population biology. This chapter provides a survey to the emerging science of collectives.

### Citations

3773 | Reinforcement Learning: An Introduction
- Sutton, Barto
- 1998
(Show Context)
Citation Context ...tool for the types of problems addressed by collectives. The goal of an RL algorithm is to determine how, using those reward signals, the agent should update its action policy to maximize its utility =-=[123, 220, 221, 232]-=-. Because RL generally provides model-free 7 and “online” learning features, it is ideally suited for the distributed environment where a “teacher” is not available and the agents need to learn succes... |

2295 |
Game Theory
- Fudenberg, D, et al.
- 1991
(Show Context)
Citation Context ...alled ”agents”, along a world utility function that measures the full system’s performance 4 . Many fields provide partial solutions to the design and study of collectives. In particular, game theory =-=[11, 19, 30, 87]-=-, mechanism design [82, 87], and multiagent reinforcement learning [53, 56, 112, 192] are fields that grapple with some of the issues encountered in the field of collectives. details). However, though... |

1788 |
Data Networks
- Bertsekas, Gallagher
- 1991
(Show Context)
Citation Context ... properties like competition, cooperation and collective behavior and especially their relationship to market efficiency [58, 122, 197]. 4.2 Data Routing in a Network Packet routing in a data network =-=[28, 110, 212, 231, 127, 94]-=- presents a particularly interesting domain for the investigation of collectives. In particular, with such routing: (i) the problem is inherently distributed; (ii) for all but the most trivial network... |

1363 | Intelligence without representation
- Brooks
- 1991
(Show Context)
Citation Context ...(e.g. reasoning, understanding, learning etc.) rather than on their cumulative character. As the idea that intelligence may have more to do with the interaction among components started to take shape =-=[41, 42]-=-, focus shifted to concepts (e.g., multi-agent systems) that better incorporated that idea [121]. Multi-Agent Systems The field of Multi-Agent Systems (MAS) is concerned with the interactions among th... |

1303 | Reinforcement Learning: A Survey
- Kaelbling, Littman, et al.
- 1996
(Show Context)
Citation Context ...tool for the types of problems addressed by collectives. The goal of an RL algorithm is to determine how, using those reward signals, the agent should update its action policy to maximize its utility =-=[123, 220, 221, 232]-=-. Because RL generally provides model-free 7 and “online” learning features, it is ideally suited for the distributed environment where a “teacher” is not available and the agents need to learn succes... |

1227 | Learning to predict by the methods of temporal differences
- Sutton
- 1988
(Show Context)
Citation Context ...tool for the types of problems addressed by collectives. The goal of an RL algorithm is to determine how, using those reward signals, the agent should update its action policy to maximize its utility =-=[123, 220, 221, 232]-=-. Because RL generally provides model-free 7 and “online” learning features, it is ideally suited for the distributed environment where a “teacher” is not available and the agents need to learn succes... |

1208 |
Counterspeculation, Auctions, and Competitive Sealed Tenders
- Vickrey
- 1961
(Show Context)
Citation Context ...auctions is the subject of the field of mechanism design. Broadly defined, mechanism design is concerned with the incentives that must be applied to any set of agents that interact and exchange goods =-=[87, 164, 229]-=- in order to get those agents to exhibit desired behavior. Usually the desired behavior concerns pre-specified ‘inherent’ utility functions of some sort for each of the individual agents. In particula... |

779 |
The theory of Learning in Games
- Fudenburg, Levine
- 1998
(Show Context)
Citation Context ...nt to modify its strategy from one game to the next by “learning” from its memory of past games, in a bounded rational manner. The field of learning in games is concerned with exactly such situations =-=[86, 12, 17, 26, 70, 126, 178, 173]-=-. Most of the formal work in this field involves simple models for the learning process of the agents. For example, in ‘fictitious play’ [86], in each successive game, each agent i adopts what would b... |

778 |
Evolution and the Theory of Games
- Smith, J
- 1982
(Show Context)
Citation Context ...e individual agents can be taken to be their reproductive fitness. There is an entire subfield of game theory concerned with this connection with population biology, called ‘evolutionary game theory’ =-=[155, 157]-=-. To introduce evolutionary game theory, consider a game in which all players share the same space of possible strategies, and there is an additional space of possible ‘attribute vectors’ that charact... |

757 | Intelligence Without Reason
- Brooks
- 1991
(Show Context)
Citation Context ...(e.g. reasoning, understanding, learning etc.) rather than on their cumulative character. As the idea that intelligence may have more to do with the interaction among components started to take shape =-=[41, 42]-=-, focus shifted to concepts (e.g., multi-agent systems) that better incorporated that idea [121]. Multi-Agent Systems The field of Multi-Agent Systems (MAS) is concerned with the interactions among th... |

727 |
The Evolution of Cooperation. Basic
- Axelrod
- 1984
(Show Context)
Citation Context ...nt to modify its strategy from one game to the next by “learning” from its memory of past games, in a bounded rational manner. The field of learning in games is concerned with exactly such situations =-=[86, 12, 17, 26, 70, 126, 178, 173]-=-. Most of the formal work in this field involves simple models for the learning process of the agents. For example, in ‘fictitious play’ [86], in each successive game, each agent i adopts what would b... |

645 | No free lunch theorems for optimization
- Wolpert, Macready
- 1997
(Show Context)
Citation Context ...ng are concerned with the large-scale “emergent” processes that govern the systems that consist of many (relatively) simple entities interacting with one another [24, 99]. As usually cast, the 10 See =-=[150, 236]-=- though for some counter-arguments to the particular claims most commonly made in this regard.sA Survey of Collectives 19 “simple entities” are members of one or more species, and the interactions are... |

636 |
The Society of Mind
- Minsky
- 1985
(Show Context)
Citation Context ...hat is related to the design of collectives. Indeed, one of the most famous speculative works in the field can be viewed as an argument that AI should be approached as a design of collectives problem =-=[163]-=-. Below, we discuss some topics relevant to collectives from this domain. Distributed Artificial Intelligence The field of Distributed Artificial Intelligence (DAI) has arisen as more and more traditi... |

633 | Ant colony system: a cooperative learning approach to the travelling salesman problem
- Dorigo, Gambardella
- 1997
(Show Context)
Citation Context ...selection. (See the discussion on this topic in the subsection on ecological modeling.) Swarm intelligences have been used to adaptively allocate tasks [33, 136], solve the traveling salesman problem =-=[62, 63]-=- and route data efficiently in dynamic networks [32, 201, 219] among others. However, there is no general framework for adapting swarm intelligences to maximize particular world utility functions. Acc... |

633 |
Equilibrium points in n-person games
- Nash
- 1950
(Show Context)
Citation Context ... equilibrium, or many equilibria in the space of pure strategies. A beautiful and seminal theorem due to Nash proves that every game has at least one Nash equilibrium in the space of mixed strategies =-=[171]-=-. There are several different reasons one might expect a game to result in a Nash equilibrium. One is that it is the point that perfectly rational Bayesian agents would adopt, assuming the probability... |

617 |
Handbook of Stochastic Methods
- Gardiner
- 1985
(Show Context)
Citation Context ... This work involves many aspects of Monte Carlo numerical algorithms [172], all of Markov Chains [80, 177, 215], and especially Markov fields, a topic that encompasses the ChapmanKolmogorov equations =-=[91]-=- and its variants: Liouville’s equation, the Fokker-Plank equation, and the Detailed-balance equation in particular. Non-linear dynamics issA Survey of Collectives 23 also related to this body of work... |

613 | Some studies in machine learning using the game of checkers
- Samuel
- 1959
(Show Context)
Citation Context ... from the overall system at various intervals. It is even possible for the learners to use those rewards to modify how they learn [199, 200]. Although work on RL dates back to Samuel’s checker player =-=[191]-=-, relatively recent theoretical [232] and empirical results [56, 224] have made RL one of the most active areas in machine learning. Many problems ranging from controlling a robot’s gait to controllin... |

608 | Bayesian Learning for Neural Networks
- Neal
- 1996
(Show Context)
Citation Context ... systems for which the probability distribution over possible future states conditioned on preceding states is explicitly provided. This work involves many aspects of Monte Carlo numerical algorithms =-=[172]-=-, all of Markov Chains [80, 177, 215], and especially Markov fields, a topic that encompasses the ChapmanKolmogorov equations [91] and its variants: Liouville’s equation, the Fokker-Plank equation, an... |

586 |
L.: Rete: A fast algorithm for the many pattern/many object pattern match problem
- Forgy
- 1982
(Show Context)
Citation Context ...ence (AI) tasks have migrated toward parallel implementation. The most direct approach to such implementations is to directly parallelize AI production systems or the underlying programming languages =-=[79, 189]-=-. An alternative and more challenging approach is to use distributed computing, where not only are the individual reasoning, planning and scheduling AI tasks parallelized, but there are different modu... |

580 |
Introduction to the Numerical Solution of Markov Chains
- Stewart
- 1994
(Show Context)
Citation Context ...bility distribution over possible future states conditioned on preceding states is explicitly provided. This work involves many aspects of Monte Carlo numerical algorithms [172], all of Markov Chains =-=[80, 177, 215]-=-, and especially Markov fields, a topic that encompasses the ChapmanKolmogorov equations [91] and its variants: Liouville’s equation, the Fokker-Plank equation, and the Detailed-balance equation in pa... |

556 | 2001): “Algorithmic mechanism design - Nisan, Ronen |

498 | Markov games as a framework for multi-agent reinforcement learning - Littman - 1994 |

460 |
Olsder.Dynamic Noncooperative Game Theory
- Basar, J
- 1999
(Show Context)
Citation Context ...alled ”agents”, along a world utility function that measures the full system’s performance 4 . Many fields provide partial solutions to the design and study of collectives. In particular, game theory =-=[11, 19, 30, 87]-=-, mechanism design [82, 87], and multiagent reinforcement learning [53, 56, 112, 192] are fields that grapple with some of the issues encountered in the field of collectives. details). However, though... |

453 | Locally weighted learning
- Atkeson, Moore, et al.
- 1997
(Show Context)
Citation Context ... provides the potential laws that allow order to arise from chaos [125]. Adaptive Control Theory Adaptive control [7, 196], and in particular adaptive control involving locally weighted RL algorithms =-=[9, 165]-=-, constitute a broadly applicable framework for controlling small, potentially inexactly modeled systems. Augmented by techniques in the control of chaotic systems [52, 60, 61], they constitute a very... |

424 |
Growing Artificial Societies: Social Science from the Bottom Up
- Epstein, Axtell
- 1996
(Show Context)
Citation Context ...ogical modeling can be applied to other non-biological systems. In particular, those models shed light on social issues such as the emergence of language or culture, warfare, and economic competition =-=[71, 72, 88]-=-. They also can be used to investigate more abstract issues concerning the behavior of large complex systems with many interacting components [89, 98, 156, 176, 184]. Going a bit further afield, an ap... |

368 | Practical issues in temporal difference learning
- Tesauro
- 1992
(Show Context)
Citation Context ...e for the learners to use those rewards to modify how they learn [199, 200]. Although work on RL dates back to Samuel’s checker player [191], relatively recent theoretical [232] and empirical results =-=[56, 224]-=- have made RL one of the most active areas in machine learning. Many problems ranging from controlling a robot’s gait to controlling a chemical plant to allocating constrained resource have been addre... |

351 |
Adaptive control
- Astrom, Wittenmark
- 1995
(Show Context)
Citation Context ...p between complexity and self-organization is a particularly important one, in that it provides the potential laws that allow order to arise from chaos [125]. Adaptive Control Theory Adaptive control =-=[7, 196]-=-, and in particular adaptive control involving locally weighted RL algorithms [9, 165], constitute a broadly applicable framework for controlling small, potentially inexactly modeled systems. Augmente... |

336 |
The complexity of cooperation: Agent-based models of competition and collaboration
- Axelrod
- 1997
(Show Context)
Citation Context ...s rarely trivial; one of the main difficulties encountered in MAS design is that agents act selfishly and artificial cooperation structures have to be imposed on their behavior to enforce cooperation =-=[13]-=-. An active area of research, which holds promise for addressing parts the design of collectives problem, is to determine how selfish agents’ “incentives” have to be engineered in order to avoid probl... |

314 |
Markov chains
- Norris
- 1998
(Show Context)
Citation Context ...bility distribution over possible future states conditioned on preceding states is explicitly provided. This work involves many aspects of Monte Carlo numerical algorithms [172], all of Markov Chains =-=[80, 177, 215]-=-, and especially Markov fields, a topic that encompasses the ChapmanKolmogorov equations [91] and its variants: Liouville’s equation, the Fokker-Plank equation, and the Detailed-balance equation in pa... |

304 | The dynamics of reinforcement learning in cooperative multiagent systems
- Claus, Boutilier
- 1998
(Show Context)
Citation Context ...ormance 4 . Many fields provide partial solutions to the design and study of collectives. In particular, game theory [11, 19, 30, 87], mechanism design [82, 87], and multiagent reinforcement learning =-=[53, 56, 112, 192]-=- are fields that grapple with some of the issues encountered in the field of collectives. details). However, though these fields provide some of the ingredients required for a full fledged field of co... |

294 | Multiagent systems: A survey from a machine learning perspective. Autonomous Robots - Stone, Veloso - 2000 |

294 | The Evolution of Conventions - Young - 1993 |

293 | Correlated equilibrium as an expression of Bayesian rationality
- Aumann
- 1987
(Show Context)
Citation Context ...brium. One is that it is the point that perfectly rational Bayesian agents would adopt, assuming the probability distributions they used to calculate expected payoffs were consistent with one another =-=[10, 124]-=-. A related reason, arising even in a non-Bayesian setting, is that a Nash equilibrium provides “consistent” predictions, in that if all parties predict that the game will converge to a Nash equilibri... |

283 | M.P.: Multiagent reinforcement learning: Theoretical framework and an algorithm
- Hu, Wellman
- 1998
(Show Context)
Citation Context ...ormance 4 . Many fields provide partial solutions to the design and study of collectives. In particular, game theory [11, 19, 30, 87], mechanism design [82, 87], and multiagent reinforcement learning =-=[53, 56, 112, 192]-=- are fields that grapple with some of the issues encountered in the field of collectives. details). However, though these fields provide some of the ingredients required for a full fledged field of co... |

282 |
At home in the universe: The search for the laws of self-organization and complexity
- Kauffman
- 1995
(Show Context)
Citation Context ... case of self-organization [59]. The relationship between complexity and self-organization is a particularly important one, in that it provides the potential laws that allow order to arise from chaos =-=[125]-=-. Adaptive Control Theory Adaptive control [7, 196], and in particular adaptive control involving locally weighted RL algorithms [9, 165], constitute a broadly applicable framework for controlling sma... |

279 | Improving elevator performance using reinforcement learning
- Crites, Barto
- 1996
(Show Context)
Citation Context ...ormance 4 . Many fields provide partial solutions to the design and study of collectives. In particular, game theory [11, 19, 30, 87], mechanism design [82, 87], and multiagent reinforcement learning =-=[53, 56, 112, 192]-=- are fields that grapple with some of the issues encountered in the field of collectives. details). However, though these fields provide some of the ingredients required for a full fledged field of co... |

275 | Acting optimally in partially observable stochastic domains
- Cassandra, Kaelbling, et al.
- 1994
(Show Context)
Citation Context ... powerful framework for analyzing both biological and economic systems. For example, natural selection (particularly punctuated 12 In contrast, in the field of Markov decision processes, discussed in =-=[45]-=-, the full system may be a Markov field, but the system designer only sets the conditional transition probability distribution of a few of the field elements at most, to the appropriate “decision rule... |

274 | A market-oriented programming environment and its application to distributed multicommodity flow problems
- Wellman
- 1993
(Show Context)
Citation Context ...uits [75], “stealing” unused CPU cycles in a network of computers [69, 230], predicting option futures in financial markets [185], and numerous scheduling and distributed resource allocation problems =-=[138, 142, 210, 218, 234, 235]-=-. Computational economics can also be used for tasks not tightly coupled to resource allocation. For example, following the work of Maes [151] and Ferber [74], Baum shows how by using computational ma... |

258 | Pricing congestible network resources
- MacKie-Mason, Varian
- 1995
(Show Context)
Citation Context ...tial alternatives to current routing schemes [25, 67, 132, 133, 139, 141, 179, 180, 208]. Closely related is work on various “pricing”based resource allocation strategies in congestable data networks =-=[149]-=-. This work is at least partially based upon current understanding of pricing in toll lanes, and traffic flow in general (see below). All of these approaches are particularly of interest when combined... |

257 | Evolution of indirect reciprocity by image scoring.” Nature
- Nowak, Sigmund
- 1998
(Show Context)
Citation Context ...nt to modify its strategy from one game to the next by “learning” from its memory of past games, in a bounded rational manner. The field of learning in games is concerned with exactly such situations =-=[86, 12, 17, 26, 70, 126, 178, 173]-=-. Most of the formal work in this field involves simple models for the learning process of the agents. For example, in ‘fictitious play’ [86], in each successive game, each agent i adopts what would b... |

246 | Spawn: A distributed computational economy
- Waldspurger, Hogg, et al.
- 1992
(Show Context)
Citation Context ...ains where market mechanisms were successfully applied include purchasing memory in an operating systems [50], allocating virtual circuits [75], “stealing” unused CPU cycles in a network of computers =-=[69, 230]-=-, predicting option futures in financial markets [185], and numerous scheduling and distributed resource allocation problems [138, 142, 210, 218, 234, 235]. Computational economics can also be used fo... |

225 | Learning in extensive-form games: Experimental data and simple dynamic models in the intermediate term
- Roth, Erev
- 1995
(Show Context)
Citation Context ...such strategies that i has encountered in the past. More sophisticated versions of this work employ simple Bayesian learning algorithms, or re-inventions of some of the techniques of the RL community =-=[190]-=-. Typically in learning in games one defines a payoff to the agent for a sequence of games, for example as a discounted sum of the payoffs in each of the constituent games. Within this framework one c... |

225 |
The roles of mutation, inbreeding, crossbreeding and selection in evolution
- Wright
- 1932
(Show Context)
Citation Context ...arning algorithms. However it has long been appreciated that there are many ways in which viewing biological systems as involving searches over such functions can lead to deeper understanding of them =-=[203, 244]-=-. Conversely, some have argued that the mechanism underlying biological systems can be used to help design search algorithms [109]. 10 These kinds of reasoning which relate utility functions and biolo... |

223 | Competitive Markov Decision Processes - Filar, Vrieze - 1997 |

215 | Rational Learning leads to Nash Equilibrium
- Kalain, Lehrer
- 1993
(Show Context)
Citation Context ...brium. One is that it is the point that perfectly rational Bayesian agents would adopt, assuming the probability distributions they used to calculate expected payoffs were consistent with one another =-=[10, 124]-=-. A related reason, arising even in a non-Bayesian setting, is that a Nash equilibrium provides “consistent” predictions, in that if all parties predict that the game will converge to a Nash equilibri... |

214 | Issues in Automated Negotiation and Electronic Commerce: Extending the Contract Net Framework
- Sandholm, Lesser
- 1995
(Show Context)
Citation Context ...traditional AI fashion often results in brittle solutions. Accordingly, recently there has been a move toward both more autonomous modules and fewer restrictions on the interactions among the modules =-=[194]-=-. Despite this evolution, DAI maintains the traditional AI concern with a prefixed set of particular aspects of intelligent behavior (e.g. reasoning, understanding, learning etc.) rather than on their... |

208 | Coalition structure generation with worst case guarantees. AI, 111(1–2):pages 209–238
- Sandholm, Larson, et al.
- 1999
(Show Context)
Citation Context ...ncentives is not sufficient, one can resort to strategies that actively induce agents to cooperate rather than act selfishly. In such cases coordination [205], negotiations [135], coalition formation =-=[193, 195, 249]-=- or contracting [3] among agents may be needed to ensure that they do not work at cross purposes. Unfortunately, all of these approaches share with DAI and its offshoots the problem of relying on hand... |

203 |
Self-organised criticality
- Bak, Tang, et al.
- 1988
(Show Context)
Citation Context ...e able to “paint” an airplane wing with such matter and have it improve drag and lift properties significantly. Self Organizing Systems The concept of self-organization and self-organized criticality =-=[15]-=- was originally developed to help understand why many distributed physical systems are attracted to critical states that possess long-range dynamic correlations in the large-scale characteristics of t... |

197 | Dynamics of Complex Systems - Bar-Yam - 1997 |

196 |
Elements for a theory of teams
- Marschak
- 1955
(Show Context)
Citation Context ...t if one considers mechanism design in the limiting case of no restrictions on γ(.), the associated “mechanism design solution” of a team game often will result in poor performance [238]. Team theory =-=[105, 153]-=- is one approach that has been tried to circumvent this problem. The idea there is to remove all notions of a private or inherent utility, and solve directly for the strategy profile that will maximiz... |