Results 1  10
of
35
Cooperative MultiAgent Learning: The State of the Art
 Autonomous Agents and MultiAgent Systems
, 2005
"... Cooperative multiagent systems are ones in which several agents attempt, through their interaction, to jointly solve tasks or to maximize utility. Due to the interactions among the agents, multiagent problem complexity can rise rapidly with the number of agents or their behavioral sophistication. ..."
Abstract

Cited by 182 (8 self)
 Add to MetaCart
(Show Context)
Cooperative multiagent systems are ones in which several agents attempt, through their interaction, to jointly solve tasks or to maximize utility. Due to the interactions among the agents, multiagent problem complexity can rise rapidly with the number of agents or their behavioral sophistication. The challenge this presents to the task of programming solutions to multiagent systems problems has spawned increasing interest in machine learning techniques to automate the search and optimization process. We provide a broad survey of the cooperative multiagent learning literature. Previous surveys of this area have largely focused on issues common to specific subareas (for example, reinforcement learning or robotics). In this survey we attempt to draw from multiagent learning work in a spectrum of areas, including reinforcement learning, evolutionary computation, game theory, complex systems, agent modeling, and robotics. We find that this broad view leads to a division of the work into two categories, each with its own special issues: applying a single learner to discover joint solutions to multiagent problems (team learning), or using multiple simultaneous learners, often one per agent (concurrent learning). Additionally, we discuss direct and indirect communication in connection with learning, plus open issues in task decomposition, scalability, and adaptive dynamics. We conclude with a presentation of multiagent learning problem domains, and a list of multiagent learning resources. 1
Theoretical advantages of lenient learners: An evolutionary game theoretic perspective
 Journal of Machine Learning Research
"... This paper presents the dynamics of multiple learning agents from an evolutionary game theoretic perspective. We provide replicator dynamics models for cooperative coevolutionary algorithms and for traditional multiagent Qlearning, and we extend these differential equations to account for lenient l ..."
Abstract

Cited by 23 (12 self)
 Add to MetaCart
(Show Context)
This paper presents the dynamics of multiple learning agents from an evolutionary game theoretic perspective. We provide replicator dynamics models for cooperative coevolutionary algorithms and for traditional multiagent Qlearning, and we extend these differential equations to account for lenient learners: agents that forgive possible mismatched teammate actions that resulted in low rewards. We use these extended formal models to study the convergence guarantees for these algorithms, and also to visualize the basins of attraction to optimal and suboptimal solutions in two benchmark coordination problems. The paper demonstrates that lenience provides learners with more accurate information about the benefits of performing their actions, resulting in higher likelihood of convergence to the globally optimal solution. In addition, the analysis indicates that the choice of learning algorithm has an insignificant impact on the overall performance of multiagent learning algorithms; rather, the performance of these algorithms depends primarily on the level of lenience that the agents exhibit to one another. Finally, the research herein supports the strength and generality of evolutionary game theory as a backbone for multiagent learning.
A sensitivity analysis of a cooperative coevolutionary algorithm biased for optimization
, 2004
"... Recent theoretical work helped explain certain optimizationrelated pathologies in cooperative coevolutionary algorithms (CCEAs). Such explanations have led to adopting specific and constructive strategies for improving CCEA optimization performance by biasing the algorithm toward ideal collaboratio ..."
Abstract

Cited by 18 (7 self)
 Add to MetaCart
(Show Context)
Recent theoretical work helped explain certain optimizationrelated pathologies in cooperative coevolutionary algorithms (CCEAs). Such explanations have led to adopting specific and constructive strategies for improving CCEA optimization performance by biasing the algorithm toward ideal collaboration. This paper investigates how sensitivity to the degree of bias (set in advance) is affected by certain algorithmic and problem properties. We discover that the previous static biasing approach is quite sensitive to a number of problem properties, and we propose a stochastic alternative which alleviates this problem. We believe that finding appropriate biasing rates is more feasible with this new biasing technique.
Evolving Policy Geometry for Scalable Multiagent Learning
, 2010
"... A major challenge for traditional approaches to multiagent learning is to train teams that easily scale to include additional agents. The problem is that such approaches typically encode each agent’s policy separately. Such separation means that computational complexity explodes as the numberofagent ..."
Abstract

Cited by 15 (7 self)
 Add to MetaCart
(Show Context)
A major challenge for traditional approaches to multiagent learning is to train teams that easily scale to include additional agents. The problem is that such approaches typically encode each agent’s policy separately. Such separation means that computational complexity explodes as the numberofagentsintheteamincreases,andalsoleadstothe problem of reinvention: Skills that should be shared among agents must be rediscovered separately for each agent. To address this problem, this paper presents an alternative evolutionary approach to multiagent learning called multiagent HyperNEAT that encodes the team as a pattern of related policies rather than as a set of individual agents. To capture this pattern, a policy geometry is introduced to describe the relationship between each agent’s policy and its canonical geometric position within the team. Because policy geometry can encode variations of a shared skill across all of the policies it represents, the problem of reinvention is avoided. Furthermore, because the policy geometry of a particular team can be sampled at any resolution, it acts as a heuristic for generating policies for teams of any size, producing a powerful new capability for multiagent learning. In this paper, multiagent HyperNEAT is tested in predatorprey and roomclearing domains. In both domains the results are effective teams that can be successfully scaled to larger team sizes without any further training. Categories and Subject Descriptors: I.2.6 [Artificial Intelligence]: Learning—connectionism and neural nets, concept learning;I.2.11[Artificial Intelligence]:
A visual demonstration of convergence properties of cooperative coevolution
 IN PARALLEL PROBLEM SOLVING FROM NATURE – PPSN2004
, 2004
"... We introduce a model for cooperative coevolutionary algorithms (CCEAs) using partial mixing, which allows us to compute the expected longrun convergence of such algorithms when individuals ’ fitness is based on the maximum payoff of some N evaluations with partners chosen at random from the other ..."
Abstract

Cited by 14 (10 self)
 Add to MetaCart
We introduce a model for cooperative coevolutionary algorithms (CCEAs) using partial mixing, which allows us to compute the expected longrun convergence of such algorithms when individuals ’ fitness is based on the maximum payoff of some N evaluations with partners chosen at random from the other population. Using this model, we devise novel visualization mechanisms to attempt to qualitatively explain a difficulttoconceptualize pathology in CCEAs: the tendency for them to converge to suboptimal Nash equilibria. We further demonstrate visually how increasing the size of N, or biasing the fitness to include an idealcollaboration factor, both improve the likelihood of optimal convergence, and under which initial population configurations they are not much help.
Archivebased cooperative coevolutionary algorithms, GECCO ’06
 Proceedings of the 8th annual conference on Genetic and evolutionary computation
, 2006
"... Archivebased cooperative coevolutionary algorithms attempt to retain a set of individuals which act as good collaborators for other coevolved individuals in the evolutionary system. We introduce a new archivebased algorithm, called iCCEA, which compares favorably with other cooperative coevolution ..."
Abstract

Cited by 14 (2 self)
 Add to MetaCart
(Show Context)
Archivebased cooperative coevolutionary algorithms attempt to retain a set of individuals which act as good collaborators for other coevolved individuals in the evolutionary system. We introduce a new archivebased algorithm, called iCCEA, which compares favorably with other cooperative coevolutionary algorithms. We explain the current problems with cooperative coevolution which have given rise to archive methods, detail the iCCEA algorithm, compare it against other traditional and archivebased methods on basic problem domains, and discuss the reasons behind the performance of various algorithms.
Robustness in Cooperative Coevolution
, 2006
"... Though recent analysis of traditional cooperative coevolutionary algorithms (CCEAs) casts doubt on their suitability for static optimization tasks, our experience is that the algorithms perform quite well in multiagent learning settings. This is due in part because many CCEAs may be quite suitable t ..."
Abstract

Cited by 11 (0 self)
 Add to MetaCart
(Show Context)
Though recent analysis of traditional cooperative coevolutionary algorithms (CCEAs) casts doubt on their suitability for static optimization tasks, our experience is that the algorithms perform quite well in multiagent learning settings. This is due in part because many CCEAs may be quite suitable to finding behaviors for team members that result in good (though not necessarily optimal) performance but which are also robust to changes in other team members. Given this, there are two main goals of this paper. First, we describe a general framework for clearly defining robustness, offering a specific definition for our studies. Second, we examine the hypothesis that CCEAs exploit this robustness property during their search. We use an existing theoretical model to gain intuition about the kind of problem properties that attract populations in the system, then provide a simple empirical study justifying this intuition in a practical setting. The results are the first steps toward a constructive view of CCEAs as optimizers of robustness.
Coevolution of Heterogeneous MultiRobot Teams
 Proceedings of the Genetic and Evolutionary Computation Conference
, 2010
"... Evolving multiple robots so that each robot acting independently can contribute to the maximization of a system level objective presents significant scientific challenges. For example, evolving multiple robots to maximize aggregate information in exploration domains (e.g., planetary exploration, sea ..."
Abstract

Cited by 9 (8 self)
 Add to MetaCart
(Show Context)
Evolving multiple robots so that each robot acting independently can contribute to the maximization of a system level objective presents significant scientific challenges. For example, evolving multiple robots to maximize aggregate information in exploration domains (e.g., planetary exploration, search and rescue) requires coordination, which in turn requires the careful design of the evaluation functions. Additionally, where communication among robots is expensive (e.g., limited power or computation), the coordination must be achieved passively, without robots explicitly informing others of their states/intended actions. Coevolving robots in these situations is a potential solution to producing coordinated behavior, where the robots are coupled through their evaluation functions. In this work, we investigate coevolution in three types of domains: (i) where precisely n homogeneous robots need to perform a task; (ii) where n is the optimal number of homogeneous robots for the task; and (iii) where n is the optimal number of heterogeneous robots for the task. Our results show that coevolving robots with evaluation functions that are locally aligned with the system evaluation significantly improve performance over robots evolving using the system evaluation function directly, particularly in dynamic environments.
Avoiding convergence in cooperative coevolution with novelty search
 In International Conference on Autonomous Agents & Multiagent Systems (AAMAS
, 2014
"... Cooperative coevolution is an approach for evolving solutions composed of coadapted components. Previous research has shown, however, that cooperative coevolutionary algorithms are biased towards stability: they tend to converge prematurely to equilibrium states, instead of converging to optimal o ..."
Abstract

Cited by 8 (8 self)
 Add to MetaCart
(Show Context)
Cooperative coevolution is an approach for evolving solutions composed of coadapted components. Previous research has shown, however, that cooperative coevolutionary algorithms are biased towards stability: they tend to converge prematurely to equilibrium states, instead of converging to optimal or nearoptimal solutions. In singlepopulation evolutionary algorithms, novelty search has been shown capable of avoiding premature convergence to local optima — a pathology similar to convergence to equilibrium states. In this study, we demonstrate how novelty search can be applied to cooperative coevolution by proposing two new algorithms. The first algorithm promotes behavioural novelty at the team level (NST), while the second promotes novelty at the individual agent level (NSI). The proposed algorithms are evaluated in two popular multiagent tasks: predatorprey pursuit and keepaway soccer. An analysis of the explored collaboration space shows that (i) fitnessbased evolution tends to quickly converge to poor equilibrium states, (ii) NSI almost never reaches any equilibrium state due to constant change in the individual populations, while (iii) NST explores a variety of equilibrium states in each evolutionary run and thus significantly outperforms both fitnessbased evolution and NSI.
Theoretical convergence guarantees for cooperative coevolutionary algorithms
 Evolutionary Computation
"... Cooperative coevolutionary algorithms have the potential to significantly speed up the search process by dividing the space into parts that can each be conquered separately. However, recent research presented theoretical and empirical arguments that these algorithms tend to converge to suboptimal s ..."
Abstract

Cited by 8 (0 self)
 Add to MetaCart
(Show Context)
Cooperative coevolutionary algorithms have the potential to significantly speed up the search process by dividing the space into parts that can each be conquered separately. However, recent research presented theoretical and empirical arguments that these algorithms tend to converge to suboptimal solutions in the search space, and are thus not fit for optimization tasks. This paper details an extended formal model for cooperative coevolutionary algorithms, and uses it to explore possible reasons these algorithms converge to optimal or suboptimal solutions. We demonstrate that, under specific conditions, this theoretical model will converge to the globally optimal solution. The proofs provide the underlying theoretical foundation for a better application of cooperative coevolutionary algorithms. We demonstrate the practical advantages of applying ideas from this theoretical work to a simple problem domain.