This paper addresses the evolution of control strategies for a collective: a set of entities that collectively strives to maximize a global evaluation function that rates the performance of the full system. Directly addressing such problems by having a population of collectives and applying the evolutionary algorithm to that population is appealing, but the search space is prohibitively large in most cases. Instead, we focus on evolving control policies for each member of the collective. The fundamental issue in this approach is how to create an evaluation function for each member of the collective that is both aligned with the global evaluation function and is sensitive to the fitness changes of the member, while relatively insensitive to the fitness changes of other members. We show how to construct evaluation functions in dynamic, noisy and communication-limited collective environments. On a rover coordination problem, a control policy evolved using aligned and member-sensitive evaluations outperforms global evaluation methods by up to 400%. More notably, in the presence of a larger number of rovers or rovers with noisy and communication limited sensors, the proposed method outperforms global evaluation by a higher percentage than in noise-free conditions with a small number of rovers. 1.

Abstract — The performance of distributed systems generally depend on the actions and interactions of a large number of independent components (e.g., agents, neurons). Such “collectives” are often subject to communication restrictions, making it difficult for the components to coordinate their actions to provide good system level performance. In this article we address that coordination problem and derive four agent utility functions that make different tradeoffs between alignedness between agent and system utilities and the signal-to-noise each agent encounters. The results show that these utility functions outperform both traditional methods and previous collective-based methods by up to 75 % in systems with communication restrictions. I.

This paper presents a method for creating evaluation functions that efficiently promote coordination in a multiagent system, allowing single-agent evolutionary computation techniques to be extended to multi-agent domains. While this problem can be addressed directly by treating the entire multiagent system as a large single agent, the search space is prohibitively large in most cases. Instead, the proposed method focuses on having each agent use its own evolutionary computation method to maximize its own evaluation function. There are two fundamental issue in this approach: 1) how to create an evaluation function for an agent that is aligned with the global evaluation function and 2) how to create an evaluation function that is sensitive to the fitness changes of the agent, while relatively insensitive to the fitness changes of other agents. If the first issue is not addressed, the evolved agents will not coordinate well. If the second issue is not addressed, the collective evolutionary process will be inefficient and the system will be slow to converge to good solutions. This paper shows how to construct evaluation functions that resolve these issues in dynamic, noisy and communicationlimited multi-agent environments. On a rover coordination problem, a control policy evolved using aligned and member-sensitive evaluations outperforms global evaluation methods by up to 400%. More notably, in the presence of a larger number of rovers or rovers with noisy and communication limited sensors, the proposed method outperforms global evaluation by a higher percentage than in noise-free conditions with a small number of rovers.

Evolutionary methods are powerful tools in discovering solutions for difficult continuous tasks. When such a solution is encoded over multiple genes, a genetic algorithm faces the difficult credit assignment problem of evaluating how a single gene in a chromosome contributes to the full solution. Typically a single evaluation function is used for the entire chromosome, implicitly giving each gene in the chromosome the same evaluation. This method is inefficient because a gene will get credit for the contribution of all the other genes as well. Accurately measuring the fitness of individual genes in such a large search space requires many trials. This paper instead proposes turning this single complex search problem into a multi-agent search problem, where each agent has the simpler task of discovering a suitable gene. Gene-specific evaluation functions can then be created that have better theoretical properties than a single evaluation function over all genes. This method is tested in the difficult double-pole balancing problem, showing that agents using gene-specific evaluation functions can create a successful control policy in 20 % fewer trials than the best existing genetic algorithms. The method is extended to more distributed problems, achieving 95 % performance gains over tradition methods in the multi-rover domain. 1.

Summary. Coordinating the behavior of a large number of agents to achieve a system level goal poses unique design challenges. In particular, problems of scaling (number of agents in the thousands to tens of thousands), observability (agents have limited sensing capabilities), and robustness (the agents are unreliable) make it impossible to simply apply methods developed for small multi-agent systems composed of reliable agents. To address these problems, we present an approach based on deriving agent goals that are aligned with the overall system goal, and can be computed using information readily available to the agents. Then, each agent uses a simple reinforcement learning algorithm [26] to pursue its own goals. Because of the way in which those goals are derived, there is no need to use difficult to scale external mechanisms to force collaboration or coordination among the agents, or to ensure that agents actively attempt to appropriate the tasks of agents that suffered failures. To present these results in a concrete setting, we focus on the problem of finding the subset of a set of imperfect devices that results in the best aggregate device [5]. This is a large distributed agent coordination problem where each agent (e.g., device) needs to determine whether to be part of the aggregate device. Our results show that the approach proposed in this work provides improvements of over an order of magnitude over both traditional search methods and traditional multi-agent methods. Furthermore, the results show that even in extreme cases of agent failures (i.e., half the agents failed midway through the simulation) the system’s performance degrades gracefully and still outperforms a failure-free and centralized search algorithm. The results also show that the gains increase as the size of the system (e.g., number of agents) increases. This latter result is particularly encouraging and suggests that this method is ideally suited for domains where the number of agents is currently in the thousands and will reach tens or hundreds of thousands in the near future. 1

Chemical and biochemical systems are presented as collectives of interacting stochastic automata: each automaton represents a molecule that undergoes state transitions. This framework constitutes an artificial biochemistry, where automata interact by the equivalent of the law of mass action. We analyze several example systems and networks, both by stochastic simulation and by ordinary dif‐ ferential equations. 1 Stochastic Automata Collectives This paper is an empirical investigation of an artifi‐ cial biochemistry obtained by the interactions of sto‐ chastic automata. The study of such artificial frame‐ works has been advocated before [2]; we explore a modern version based on a theory of concurrent processes that obeys the equivalent of the law of mass action. Foundations for this work have been investigated elsewhere [1]; here we aim to give a self‐contained and accessible presentation of the framework, and to explore by means of examples the richness of “emergent ” and unexpected behavior that can be represented by combinations of simple building blocks. By a collective we mean a large set of interacting, finite state automata. This is not quite the situation we have in classical automata theory, because we are interested in the behavior of a large set of auto‐ mata acting together. It is also not quite the situation with cellular automata, because our automata are interacting, but not necessarily on a regular grid. It is also not quite the situation in process algebra, be‐ cause again we are interested in the behavior of col‐ lectives, not of individuals. Similar frameworks have been investigated under the headings of collectives [12], sometimes including stochasticity [6]. By stochastic we mean that automata interactions have rates. Stochastic rates induce a quantitative semantics for the behavior of collectives. Collective behavior cannot be considered quite discrete, be‐ cause it can be the result of hundreds or thousands individual contributions. But it is not quite continu‐ ous either, because of the possibility of non‐trivial stochastic effects. And it is also not hybrid: there is no switching between discrete and continuous re‐ gimes.

Dedicated to my parents and family.

One way to cope with the increasing demand in transportation networks is to integrate standard solutions with more intelligent measures. This paper discusses the effects of integrating co-evolving decision-making regarding route choices (by drivers) and control measures (by traffic lights) We use microscopic modeling and simulation, in opposition to the classical network analysis. General questions here are whether co-evolution pays off, and, if so, what kind of evolutionary approach shall be used. This is challenging for networks other than the two-route one due to the complexity of routechoice behavior, as well as control strategies by the traffic lights. Moreover, the more agents, the less effective learning strategies are, when the integration among them depicts complex interelationships. The approach was tested in different scenarios.

Abstract not found

We model chemical and biochemical systems as collectives of interacting stochastic automata, with each automaton representing a molecule that undergoes state transitions. In this artificial biochemistry, automata interact by the equivalent of the law of mass action. We investigate several simple but intriguing automata collectives by stochastic simulation and by ODE analysis.