In this article we describe a methodology for evolving neurocontrollers of autonomous mobile robots without human intervention. The presentation, which spans from technological and methodological issues to several experimental results on evolution of physical mobile robots, covers both previous and recent work in the attempt to provide a uni ed picture within which the reader can compare the effects of systematic variations on the experimental settings. After describing some key principles for building mobile robots and tools suitable for experiments in adaptive robotics, we give an overview of different approaches to evolutionary robotics and present our methodology. We start reviewing two basic experiments showing that different environments can shape very different behaviors and neural mechanisms under very similar selection criteria. We then address the issue of incremental evolution in two different experiments from the perspective of changing environments and robot morphologies. Finally, we investigate the possibility of evolving plastic neurocontrollers and analyze an evolved neurocontroller that relies on fast and continuously changes synapses characterized by dynamic stability. We conclude by reviewing the implications of this methodology for engineering, biology, cognitive science, and artificial life, and point at future directions of research.

such as the orientation of a line in the visual field or the location of Two main goals for reconstruction are approached in this the body in space are coded as activity levels in populations of neurons. Reconstruction or decoding is an inverse problem in which paper. The first goal is technical and is exemplified by the the physical variables are estimated from observed neural activity. population vector method applied to motor cortical activities Reconstruction is useful first in quantifying how much information during various reaching tasks (Georgopoulos et al. 1986, 1989; about the physical variables is present in the population and, second, Schwartz 1994) and the template matching method applied to in providing insight into how the brain might use distributed represen- disparity selective cells in the visual cortex (Lehky and Sejnowtations in solving related computational problems such as visual ob- ski 1990) and hippocampal place cells during rapid learning of ject recognition and spatial navigation. Two classes of reconstruction place fields in a novel environment (Wilson and McNaughton methods, namely, probabilistic or Bayesian methods and basis func- 1993). In these examples, reconstruction extracts information tion methods, are discussed. They include important existing methods from noisy neuronal population activity and transforms it to a

. A computational model of hippocampal activity during spatial cognition and navigation tasks is presented. The spatial representation in our model of the rat hippocampus is built on-line during exploration via two processing streams. An allothetic vision-based representation is built by unsupervised Hebbian learning extracting spatio-temporal properties of the environment from visual input. An idiothetic representation is learned based on internal movement-related information provided by path integration. On the level of the hippocampus, allothetic and idiothetic representations are integrated to yield a stable representation of the environment by a population of localized overlapping CA3-CA1 place fields. The hippocampal spatial representation is used as a basis for goal-oriented spatial behavior. We focus on the neural pathway connecting the hippocampus to the nucleus accumbens. Place cells drive a population of locomotor action neurons in the nucleus accumbens. Reward-based learnin...

Using experimental facts about long-term potentiation (LTP) and hippocampal place cells, we model how a spatial map of the environment can be created in the rat hippocampus. Sequential firing of place cells during exploration induces, in the model, a pattern of LTP between place cells that shifts the location coded by their ensemble activity away from the actual location of the animal. These shifts provide a navigational map that, in a simulation of the Morris maze, can guide the animal toward its goal. The model demonstrates how behaviorally generated modifications of synaptic strengths can be read out to affect subsequent behavior. Our results also suggest a way that navigational maps can be constructed from experimental recordings of hippocampal place cells. *Current address: Dept. of Brain and Cognitive Sciences, MIT E25-236, 45 Carlton St., Cambridge, MA 02139. Blockade of long term potentiation (LTP) and hippocampal lesions drastically impair

ABSTRACT: This paper presents a model of how hippocampal place cells might be used for spatial navigation in two watermaze tasks: the standard reference memory task and a delayed matching-to-place task. In the reference memory task, the escape platform occupies a single location and rats gradually learn relatively direct paths to the goal over the course of days, in each of which they perform a fixed number of trials. In the delayed matching-to-place task, the escape platform occupies a novel location on each day, and rats gradually acquire one-trial learning, i.e., direct paths on the second trial of each day. The model uses a local, incremental, and statistically efficient connectionist algorithm called temporal difference learning in two distinct components. The first is a reinforcement-based ‘‘actor-critic’ ’ network that is a general model of classical and instrumental conditioning. In this case, it is applied to navigation, using place cells to provide information about state. By itself, the actor-critic can learn the reference memory task, but this learning is inflexible to changes to the platform location. We argue that one-trial learning in the delayed matching-to-place task demands a goal-independent representation of space. This is provided by the second component of the model: a network that uses temporal difference learning and selfmotion information to acquire consistent spatial coordinates in the environment. Each component of the model is necessary at a different stage of the task; the actor-critic provides a way of transferring control to the component that performs best. The model successfully captures gradual acquisition in both tasks, and, in particular, the ultimate development of one-trial learning in the delayed matching-to-place task. Place cells report a form of stable, allocentric information that is well-suited to the various kinds of learning in the model. Hippocampus 2000;10:1–16.

In the past decade, a large number of robots has been built that explicitly implement biological navigation behaviours. We review these biomimetic approaches using a framework that allows for a common description of biological and technical navigation behaviour. The review shows that biomimetic systems make significant contributions to two fields of research: First, they provide a real world test of models of biological navigation behaviour; second, they make new navigation mechanisms available for technical applications, most notably in the field of indoor robot navigation. While simpler insect navigation behaviours have been implemented quite successfully, the more complicated way-finding capabilities of vertebrates still pose a challenge to current systems. ©2000 Elsevier Science B.V. All rights reserved.

We analyze a model of navigational map formation based on correlation-based, temporally asymmetric potentiation and depression of synapses between hippocampal place cells. We show that synaptic modification during random exploration of an environment shifts the location encoded by place cell activity in such a way that it indicates the direction from any location to a fixed target avoiding walls and other obstacles. Multiple maps to different targets can be simultaneously stored if we introduce target-dependent modulation of place cell activity. Once maps to a number of target locations in a given environment have been stored, novel maps to previously unknown target locations are automatically constructed by interpolation between existing maps.

Diverse theories of animal navigation aim at explaining how to determine and maintain a course from one place to another in the environment, although each presents a particular perspective with its own terminologies. These vocabularies sometimes overlap, but unfortunately with different meanings. This paper attempts to precisely define the existing concepts and terminologies, so as to comprehensively describe the different theories and models within the same unifying framework. We present navigation strategies within a 4 level hierarchical framework based upon levels of complexity of required processing (Guidance, Place recognition-triggered Response, Topological navigation, Metric navigation). This classification is based upon what information is perceived, represented and processed. It contrasts with common distinctions based upon availability of certain sensors or cues and rather stresses the information structure and content of central processors. We then review computat...

Our model of the spatial and temporal aspects of place cell firing, and their role in rat navigation is reviewed. The model provides a can- didate mechanism, at the level of individual cells, by which place cell information concerning self-localization could be used to guide navi- gation to previously visited reward sites. The model embodies specific predictions regarding the formation of place fields, the phase coding of place cell firing with respect to the hippocampal theta rhythm, and the formation of neuronal population vectors downstream from the place cells that code for the directions of goals during navigation. Re- cent experiments regarding the spatial distribution of place cell firing have confirmed our initial modeling hypothesis, that place fields are formed from Gaussian tuning curve inputs coding for the distances from environmental features, and enabled us to further specify the functional form of these inputs. Other recent experiments regarding the temporal distribution of place cell firing in 2-dimensional environ- ments have confirmed our predictions based on the temporal aspects of place cell firing on linear tracks. Directions for further experiments and refinements to the model are outlined for the future.

this article. Because there is no visible cue in the hidden-platform water maze task, it would not help the animal find the platform. 3. Route system. Routes stored in the hippocampus can be written out to the cortex, so that directions necessary to reach a goal are associated with local views. This is the system detailed in section 2.5 (see also section 4.3). This system requires training for each step the animal must take; it cannot learn to associate local views with directions to distant goals without hippocampal help (through route replay). The Role of the Hippocampus 97 If there were a way to show the animal the route to the goal, it might be possible to train the route system even without a hippocampus. Whishaw, Cassell, and Jarrard (1995) and Schallert, Day, Weisend, and Sutherland (1996) both showed ways to train the route system directly and found that animals could learn to solve the water maze even with hippocampal lesions. Whishaw et al. (1995) trained animals with fimbria/fornix lesions to find a visible platform and then removed the visible platform. These animals concentrated their search where the platform had been. Schallert et al. (1996) used animals with kainate-colchicine hippocampal lesions. The animals were first trained with a large platform that filled almost the entire maze. Once the animals could reach that platform reliably, it was shrunk trial by trial until it was the same size as a typical platform in a water maze task. Again, the animals could learn to solve the water maze without a hippocampus. 4.3 Where Is the Route System? Although the data are not yet conclusive, we suggest that the most likely candidate for anatomical instantiation of the route system is from posterior parietal to posterior cingulate cortex. There is a lot of evide...