Evolutionary computation is a useful technique for learning behaviors in multiagent systems. Among the several types of evolutionary computation, one natural and popular method is to coevolve multiagent behaviors in multiple, cooperating populations. Recent research has suggested that coevolutionary systems may favor stability rather than performance in some domains. In order to improve upon existing methods, this paper examines the idea of modifying traditional coevolution, biasing it to search for maximal rewards. We introduce a theoretical justification of the improved method and present experiments in three problem domains. We conclude that biasing can help coevolution find better results in some multiagent problem domains. 1

Abstract—This paper presents a first attempt at evolving the rules for a game. In contrast to almost every other paper that applies computational intelligence techniques to games, we are not generating behaviours, strategies or environments for any particular game; we are starting without a game and generating the game itself. We explain the rationale for doing this and survey the theories of entertainment and curiosity that underly our fitness function, and present the details of a simple proofof-concept experiment.

The Layered Pareto Coevolution Archive (LAPCA) was recently proposed as an effective Coevolutionary Memory (CM) which, under certain assumptions, approximates monotonic progress in coevolution. In this paper, a technique is developed that interfaces the LAPCA algorithm with NeuroEvolution of Augmenting Topologies (NEAT), a method to evolve neural networks with demonstrated efficiency in game playing domains. In addition, the behavior of LAPCA is analyzed for the first time in a complex game-playing domain: evolving neural network controllers for the game Pong. The technique is shown to keep the total number of evaluations in the order of those required by NEAT, making it applicable to complex domains. Pong players evolved with a LAPCA and with the Hall of Fame (HOF) perform equally well, but the LAPCA is shown to require significantly less space than the HOF. Therefore, combining NEAT and LAPCA is found to be an effective approach to coevolution.

Abstract — Intelligent agents in games and simulators often operate in environments subject to symmetric transformations that produce new but equally legitimate environments, such as reflections or rotations of maps. That fact suggests two hypotheses of interest for machine-learning approaches to creating intelligent agents for use in such environments. First, that exploiting symmetric transformations can broaden the range of experience made available to the agents during training, and thus result in improved performance at the task for which they are trained. Second, that exploiting symmetric transformations during training can make the agents ’ response to environments not seen during training measurably more consistent. In this paper the two hypotheses are evaluated experimentally by exploiting sensor symmetries and potential symmetries of the environment while training intelligent agents for a strategy game. The experiments reveal that when a corpus of human-generated training examples is supplemented with artificial examples generated by means of reflections and rotations, improvement is obtained in both task performance and consistency of behavior.

Much of artificial intelligence research is focused on devising optimal solutions for challenging and well-defined but highly constrained problems. However, as we begin creating autonomous agents to operate in the rich environments of modern videogames and computer simulations, it becomes important to devise agent behaviors that display the visible attributes of intelligence, rather than simply performing optimally. Such visibly intelligent behavior is difficult to specify with rules or characterize in terms of quantifiable objective functions, but it is possible to utilize human intuitions to directly guide a learning system toward the desired sorts of behavior. Policy induction from human-generated examples is a promising approach to training such agents. In this paper, such a method is developed and tested using Lamarckian neuroevolution. Artificial neural networks are evolved to control autonomous agents in a strategy game. The evolution is guided by human-generated examples of play, and the system effectively learns the policies that were used by the player to generate the examples. I.e., the agents learn visibly intelligent behavior. In the future, such methods are likely to play a central role in creating autonomous agents for complex environments, making it possible to generate rich behaviors derived from nothing more formal than the intuitively generated examples of designers, players, or subject-matter experts.

Abstract — It is sometimes useful to provide intelligent agents with some degree of stochastic behavior, particularly when used in games and simulators. The less-predictable behavior that results from the randomness can make the agents seem more believable, and would encourage the players or users to address the genuine problems presented by a game or simulator rather than simply learning to exploit the embedded agents’ predictability. However, such randomized behavior should not harm performance in the agents ’ designated tasks. This paper introduces a method, called stochastic sharpening, for training artificial neural networks as stochastic controllers for agents in discrete-state environments. Stochastic sharpening reinforces the representation of confidence values in the outputs of networks with localist encodings, and thus produces networks that recommend alternative actions on the basis of their expected utility. Such networks can be used to introduce stochastic behavior with minimal disruption of task performance, resulting in agents that are more believable and less subject to exploitation based on predictability.

Abstract — In cooperative multiagent systems, agents interact to solve tasks. Global dynamics of multiagent teams result from local agent interactions, and are complex and difficult to predict. Evolutionary computation has proven a promising approach to the design of such teams. The majority of current studies use teams composed of agents with identical control rules (“genetically homogeneous teams”) and select behavior at the team level (“team-level selection”). Here we extend current approaches to include four combinations of genetic team composition and level of selection. We compare the performance of genetically homogeneous teams evolved with individual-level selection, genetically homogeneous teams evolved with team-level selection, genetically heterogeneous teams evolved with individual-level selection, and genetically heterogeneous teams evolved with team-level selection. We use a simulated foraging task to show that the optimal combination depends on the amount of cooperation required by the task. Accordingly, we distinguish between three types of cooperative tasks and suggest guidelines for the optimal choice of genetic team composition and level of selection. Index Terms — Altruism, artificial evolution, cooperation, evolutionary robotics, fitness allocation, multiagent systems (MAS),

A major goal for AI is to allow users to interact with agents that learn in real time, making new kinds of interactive simulations, training applications, and digital entertainment possible. This paper describes such a learning technology, called real-time NeuroEvolution of Augmenting Topologies (rtNEAT), and describes how rtNEAT was used to build the NeuroEvolving Robotic Operatives (NERO) video game. This game represents a new genre of machine learning games where the player trains agents in real time to perform challenging tasks in a virtual environment. Providing laymen the capability to effectively train agents in real time with no prior knowledge of AI or machine learning has broad implications, both in promoting the field of AI and making its achievements accessible to the public at large.

Evolution of neural networks, or neuroevolution, bas been successful on many low-level control problems such as pole balancing, vehicle control, and collision warning. However, high-level strategy problems that require the integration of multiple sub-behaviors have remained difficult for neuroevolution to solve. This paper proposes the hypothesis that such problems are difficult because they are fractured: the correct action varies discontinuously as the agent moves from state to state. This hypothesis is evaluated on several examples of fractured high-level reinforcement learning domains. Standard neuroevolution methods such as NEAT indeed have difficulty solving them. However, a modification of NEAT that uses radial basis function (RBF) nodes to make precise local mutations to network output is able to do much better. These results provide a better understanding of the different types of reinforcement learning problems and the limitations of current neuroevolution methods. Thus, they lay the groundwork for creating the next generation of neuroevolution algorithms that can learn strategic high-level behavior in fractured domains.

Video games provide an opportunity and challenge for the soft computational intelligence methods like the symbolic games did for “good old-fashioned artificial intelligence. ” This article reviews the achievements and future prospects of one particular approach, that of evolving neural networks, or neuroevolution. This approach can be used to construct adaptive characters in existing video games, and it can serve as a foundation for a new genre of games based on machine learning. Evolution can be guided by human knowledge, allowing the designer to control the kinds of solutions that emerge and encouraging behaviors that appear visibly intelligent to the human player. Such techniques may allow building video games that are more engaging and entertaining than current games, and those that can serve as training environments for people. Techniques developed in these games may also be widely applicable in other fields, such as robotics, resource optimization, and intelligent assistants. 1