Results 1  10
of
34
Cooperative MultiAgent Learning: The State of the Art
 Autonomous Agents and MultiAgent Systems
, 2005
"... Cooperative multiagent systems are ones in which several agents attempt, through their interaction, to jointly solve tasks or to maximize utility. Due to the interactions among the agents, multiagent problem complexity can rise rapidly with the number of agents or their behavioral sophistication. ..."
Abstract

Cited by 182 (8 self)
 Add to MetaCart
(Show Context)
Cooperative multiagent systems are ones in which several agents attempt, through their interaction, to jointly solve tasks or to maximize utility. Due to the interactions among the agents, multiagent problem complexity can rise rapidly with the number of agents or their behavioral sophistication. The challenge this presents to the task of programming solutions to multiagent systems problems has spawned increasing interest in machine learning techniques to automate the search and optimization process. We provide a broad survey of the cooperative multiagent learning literature. Previous surveys of this area have largely focused on issues common to specific subareas (for example, reinforcement learning or robotics). In this survey we attempt to draw from multiagent learning work in a spectrum of areas, including reinforcement learning, evolutionary computation, game theory, complex systems, agent modeling, and robotics. We find that this broad view leads to a division of the work into two categories, each with its own special issues: applying a single learner to discover joint solutions to multiagent problems (team learning), or using multiple simultaneous learners, often one per agent (concurrent learning). Additionally, we discuss direct and indirect communication in connection with learning, plus open issues in task decomposition, scalability, and adaptive dynamics. We conclude with a presentation of multiagent learning problem domains, and a list of multiagent learning resources. 1
Large Scale Evolutionary Optimization Using Cooperative Coevolution
, 2007
"... Evolutionary algorithms (EAs) have been applied with success to many numerical and combinatorial optimization problems in recent years. However, they often lose their effectiveness and advantages when applied to large and complex problems, e.g., those with high dimensions. Although cooperative coevo ..."
Abstract

Cited by 64 (18 self)
 Add to MetaCart
Evolutionary algorithms (EAs) have been applied with success to many numerical and combinatorial optimization problems in recent years. However, they often lose their effectiveness and advantages when applied to large and complex problems, e.g., those with high dimensions. Although cooperative coevolution has been proposed as a promising framework for tackling highdimensional optimization problems, only limited studies were reported by decomposing a highdimensional problem into single variables (dimensions). Such methods of decomposition often failed to solve nonseparable problems, for which tight interactions exist among different decision variables. In this paper, we propose a new cooperative coevolution framework that is capable of optimizing large scale nonseparable problems. A random grouping scheme and adaptive weighting are introduced in problem decomposition and coevolution. Instead of conventional evolutionary algorithms, a novel differential evolution algorithm is adopted. Theoretical analysis is presented in this paper to show why and how the new framework can be effective for optimizing large nonseparable problems. Extensive computational studies are also carried out to evaluate the performance of newly proposed algorithm on a large number of benchmark functions with up to 1000 dimensions. The results show clearly that our framework and algorithm are effective as well as efficient for large scale evolutionary optimisation problems. We are unaware of any other evolutionary algorithms that can optimize 1000dimension nonseparable problems as effectively and efficiently as we have done.
Theoretical advantages of lenient learners: An evolutionary game theoretic perspective
 Journal of Machine Learning Research
"... This paper presents the dynamics of multiple learning agents from an evolutionary game theoretic perspective. We provide replicator dynamics models for cooperative coevolutionary algorithms and for traditional multiagent Qlearning, and we extend these differential equations to account for lenient l ..."
Abstract

Cited by 24 (12 self)
 Add to MetaCart
(Show Context)
This paper presents the dynamics of multiple learning agents from an evolutionary game theoretic perspective. We provide replicator dynamics models for cooperative coevolutionary algorithms and for traditional multiagent Qlearning, and we extend these differential equations to account for lenient learners: agents that forgive possible mismatched teammate actions that resulted in low rewards. We use these extended formal models to study the convergence guarantees for these algorithms, and also to visualize the basins of attraction to optimal and suboptimal solutions in two benchmark coordination problems. The paper demonstrates that lenience provides learners with more accurate information about the benefits of performing their actions, resulting in higher likelihood of convergence to the globally optimal solution. In addition, the analysis indicates that the choice of learning algorithm has an insignificant impact on the overall performance of multiagent learning algorithms; rather, the performance of these algorithms depends primarily on the level of lenience that the agents exhibit to one another. Finally, the research herein supports the strength and generality of evolutionary game theory as a backbone for multiagent learning.
A sensitivity analysis of a cooperative coevolutionary algorithm biased for optimization
, 2004
"... Recent theoretical work helped explain certain optimizationrelated pathologies in cooperative coevolutionary algorithms (CCEAs). Such explanations have led to adopting specific and constructive strategies for improving CCEA optimization performance by biasing the algorithm toward ideal collaboratio ..."
Abstract

Cited by 17 (7 self)
 Add to MetaCart
(Show Context)
Recent theoretical work helped explain certain optimizationrelated pathologies in cooperative coevolutionary algorithms (CCEAs). Such explanations have led to adopting specific and constructive strategies for improving CCEA optimization performance by biasing the algorithm toward ideal collaboration. This paper investigates how sensitivity to the degree of bias (set in advance) is affected by certain algorithmic and problem properties. We discover that the previous static biasing approach is quite sensitive to a number of problem properties, and we propose a stochastic alternative which alleviates this problem. We believe that finding appropriate biasing rates is more feasible with this new biasing technique.
A visual demonstration of convergence properties of cooperative coevolution
 IN PARALLEL PROBLEM SOLVING FROM NATURE – PPSN2004
, 2004
"... We introduce a model for cooperative coevolutionary algorithms (CCEAs) using partial mixing, which allows us to compute the expected longrun convergence of such algorithms when individuals ’ fitness is based on the maximum payoff of some N evaluations with partners chosen at random from the other ..."
Abstract

Cited by 15 (10 self)
 Add to MetaCart
We introduce a model for cooperative coevolutionary algorithms (CCEAs) using partial mixing, which allows us to compute the expected longrun convergence of such algorithms when individuals ’ fitness is based on the maximum payoff of some N evaluations with partners chosen at random from the other population. Using this model, we devise novel visualization mechanisms to attempt to qualitatively explain a difficulttoconceptualize pathology in CCEAs: the tendency for them to converge to suboptimal Nash equilibria. We further demonstrate visually how increasing the size of N, or biasing the fitness to include an idealcollaboration factor, both improve the likelihood of optimal convergence, and under which initial population configurations they are not much help.
Evolving Policy Geometry for Scalable Multiagent Learning
, 2010
"... A major challenge for traditional approaches to multiagent learning is to train teams that easily scale to include additional agents. The problem is that such approaches typically encode each agent’s policy separately. Such separation means that computational complexity explodes as the numberofagent ..."
Abstract

Cited by 15 (7 self)
 Add to MetaCart
(Show Context)
A major challenge for traditional approaches to multiagent learning is to train teams that easily scale to include additional agents. The problem is that such approaches typically encode each agent’s policy separately. Such separation means that computational complexity explodes as the numberofagentsintheteamincreases,andalsoleadstothe problem of reinvention: Skills that should be shared among agents must be rediscovered separately for each agent. To address this problem, this paper presents an alternative evolutionary approach to multiagent learning called multiagent HyperNEAT that encodes the team as a pattern of related policies rather than as a set of individual agents. To capture this pattern, a policy geometry is introduced to describe the relationship between each agent’s policy and its canonical geometric position within the team. Because policy geometry can encode variations of a shared skill across all of the policies it represents, the problem of reinvention is avoided. Furthermore, because the policy geometry of a particular team can be sampled at any resolution, it acts as a heuristic for generating policies for teams of any size, producing a powerful new capability for multiagent learning. In this paper, multiagent HyperNEAT is tested in predatorprey and roomclearing domains. In both domains the results are effective teams that can be successfully scaled to larger team sizes without any further training. Categories and Subject Descriptors: I.2.6 [Artificial Intelligence]: Learning—connectionism and neural nets, concept learning;I.2.11[Artificial Intelligence]:
Archivebased cooperative coevolutionary algorithms, GECCO ’06
 Proceedings of the 8th annual conference on Genetic and evolutionary computation
, 2006
"... Archivebased cooperative coevolutionary algorithms attempt to retain a set of individuals which act as good collaborators for other coevolved individuals in the evolutionary system. We introduce a new archivebased algorithm, called iCCEA, which compares favorably with other cooperative coevolution ..."
Abstract

Cited by 13 (2 self)
 Add to MetaCart
(Show Context)
Archivebased cooperative coevolutionary algorithms attempt to retain a set of individuals which act as good collaborators for other coevolved individuals in the evolutionary system. We introduce a new archivebased algorithm, called iCCEA, which compares favorably with other cooperative coevolutionary algorithms. We explain the current problems with cooperative coevolution which have given rise to archive methods, detail the iCCEA algorithm, compare it against other traditional and archivebased methods on basic problem domains, and discuss the reasons behind the performance of various algorithms.
Robustness in Cooperative Coevolution
, 2006
"... Though recent analysis of traditional cooperative coevolutionary algorithms (CCEAs) casts doubt on their suitability for static optimization tasks, our experience is that the algorithms perform quite well in multiagent learning settings. This is due in part because many CCEAs may be quite suitable t ..."
Abstract

Cited by 11 (0 self)
 Add to MetaCart
(Show Context)
Though recent analysis of traditional cooperative coevolutionary algorithms (CCEAs) casts doubt on their suitability for static optimization tasks, our experience is that the algorithms perform quite well in multiagent learning settings. This is due in part because many CCEAs may be quite suitable to finding behaviors for team members that result in good (though not necessarily optimal) performance but which are also robust to changes in other team members. Given this, there are two main goals of this paper. First, we describe a general framework for clearly defining robustness, offering a specific definition for our studies. Second, we examine the hypothesis that CCEAs exploit this robustness property during their search. We use an existing theoretical model to gain intuition about the kind of problem properties that attract populations in the system, then provide a simple empirical study justifying this intuition in a practical setting. The results are the first steps toward a constructive view of CCEAs as optimizers of robustness.
Coevolution of Heterogeneous MultiRobot Teams
 Proceedings of the Genetic and Evolutionary Computation Conference
, 2010
"... Evolving multiple robots so that each robot acting independently can contribute to the maximization of a system level objective presents significant scientific challenges. For example, evolving multiple robots to maximize aggregate information in exploration domains (e.g., planetary exploration, sea ..."
Abstract

Cited by 10 (9 self)
 Add to MetaCart
(Show Context)
Evolving multiple robots so that each robot acting independently can contribute to the maximization of a system level objective presents significant scientific challenges. For example, evolving multiple robots to maximize aggregate information in exploration domains (e.g., planetary exploration, search and rescue) requires coordination, which in turn requires the careful design of the evaluation functions. Additionally, where communication among robots is expensive (e.g., limited power or computation), the coordination must be achieved passively, without robots explicitly informing others of their states/intended actions. Coevolving robots in these situations is a potential solution to producing coordinated behavior, where the robots are coupled through their evaluation functions. In this work, we investigate coevolution in three types of domains: (i) where precisely n homogeneous robots need to perform a task; (ii) where n is the optimal number of homogeneous robots for the task; and (iii) where n is the optimal number of heterogeneous robots for the task. Our results show that coevolving robots with evaluation functions that are locally aligned with the system evaluation significantly improve performance over robots evolving using the system evaluation function directly, particularly in dynamic environments.