Shaping by successive approximations is an important animal training technique in which behavior is gradually adjusted in response to strategically timed reinforcements. We describe a computational model of this shaping process and its implementation on a mobile robot. Innate behaviors in our model are sequences of actions and enabling conditions, and shaping is a behavior editing process realized by multiple editing mechanisms. The model replicates some fundamental phenomena associated with instrumental learning in animals, and allows an RWI B21 robot to learn several distinct tasks derived from the same innate behavior. 1. Introduction Service dogs trained to assist a disabled person will respond to over 60 verbal commands to, for example, turn on lights, open a refrigerator door, or retrieve a dropped object [9]. Chicks can be taught to play a toy piano (peck out a key sequence until a reinforcement is received at the end of the tune) [6], and rats have been conditioned to perform c...

Instrumental (or operant) conditioning, a form of animal learning, is similar to reinforcement learning (Watkins, 1989) in that it allows an agent to adapt its actions to gain maximally from the environment while only being rewarded for correct performance. But animals learn much more complicated behaviors through instrumental conditioning than robots presently acquire through reinforcement learning. We describe a new computational model of the conditioning process that attempts to capture some of the aspects that are missing from simple reinforcement learning: conditioned reinforcers, shifting reinforcement contingencies, explicit action sequencing, and state space re nement. We apply our model to a task commonly used to study working memory in rats and monkeys: the DMTS (Delayed Match to Sample) task. Animals learn this task in stages. In simulation, our model also acquires the task in stages, in a similar manner. We have used the model to train an RWI B21 robot.

ABSTRACT: The very different anatomical designs of the adjacent circuitries of the cortico-hippocampal pathway, along with their somewhat different synaptic plasticity mechanisms, suggest a nearly serial pathway of distinct memory circuits each contributing its own specialized processing operation to overall hippocampal function. Modeling and formal theoretical analysis of the prominent anatomical design features of particular circuits (piriform/entorhinal cortex; hippocampal field CA3; hippocampal field CA1) are found to identify potential emergent function not readily arrived at in the absence of these formal models, and yet which once derived can be seen potentially to confer unique capabilities to an integrated hippocampal mechanism for processing memories during behavior. r 1997 Wiley-Liss, Inc. KEY WORDS: LTP; Memory; corticohippocampal pathway; CA3; CA1; dentate gyrus; entorhinal cortex; perirhinal cortex; parahippocampal cortex

The place fields of hippocampal cells in old animals sometimes change when an animal is removed from and then returned to an environment [ Barnes et al., 1997 ] . The ensemble correlation between two sequential visits to the same environment shows a strong bimodality for old animals (near 0, indicative of remapping, and greater than 0.7, indicative of a similar representation between experiences), but a strong unimodality for young animals (greater than 0.7, indicative of a similar representation between experiences). One explanation for this is the multi-map hypothesis in which multiple maps are encoded in the hippocampus: old animals may sometimes be returning to the wrong map. A theory proposed by Samsonovich and McNaughton (1997) suggests that the Barnes et al. experiment implies that the maps are pre-wired in the CA3 region of hippocampus. Here, we offer an alternative explanation in which orthogonalization properties in the dentate gyrus (DG) region of hippocampus interact with e...

Instrumental (or operant) conditioning, a form of animal learning, is similar to reinforcement learning in that it allows an agent to adapt its actions to gain maximally from the environment while only being rewarded for correct performance. But animals learn much more complicated behaviors through instrumental conditioning than robots presently acquire through reinforcement learning. We describe a new computational model of the conditioning process; our discussion focuses on a training technique called chaining. Four aspects of our model distinguishit from simple reinforcement learning: conditional reinforcers, shifting reinforcement contingencies, explicit action sequencing, and state space refinement. We apply our model to a task commonly used to study working memory in rats and monkeys: the DMTS (Delayed Match to Sample) task. Animals learn this task in stages. Our model also acquires the task in stages, in a similar manner. We have also used our learning program to control a B21 r...

Instrumental (or operant) conditioning, a form of animal learning, is similar to reinforcement learning in that it allows an agent to adapt its actions to gain maximally from the environment while only being rewarded for correct performance. But animals learn much more complicated behaviors through instrumental conditioning than robots presently acquire through reinforcement learning. We describe a new computational model of the conditioning process; our discussion focuses on a training technique called chaining. Four aspects of our model distinguishitfrom simple reinforcement learning: conditional reinforcers, shifting reinforcement contingencies, explicit action sequencing, and state space refinement. We apply our model to a task commonly used to study working memory in rats and monkeys: the DMTS (Delayed Match to Sample) task. Animals learn this task in stages. Our model also acquires the task in stages, in a similar manner. We have also used our learning program to control a B21 robot. 1

Pattern separation, pattern completion, and new neuronal codes

Sandia is a multiprogram laboratory operated by Sandia Corporation,

There has been considerable interest in the importance of oscillations in the brain and in how these oscillations relate to the firing of single neurons. Recently a number of studies have shown that the spiking of individual neurons in the medial prefrontal cortex (mPFC) become entrained to the hippocampal (HPC) theta rhythm. We recently showed that theta-entrained mPFC cells lost theta-entrainment specifically on error trials even though the firing rates of these cells did not change (Hyman et al., 2010). This implied that the level of HPC theta-entrainment of mPFC units was more predictive of trial outcome than differences in firing rates and that there is more information encoded by the mPFC on working memory tasks than can be accounted for by a simple rate code. Nevertheless, the functional meaning of mPFC entrainment to HPC theta remains a mystery. It is also unclear as to whether there are any differences in the nature of the information encoded by theta-entrained and non-entrained mPFC cells. In this review we discuss mPFC entrainment to HPC theta within the context of previous results as well as provide a more detailed analysis of the Hyman et al. (2010) data set. This re-analysis revealed that thetaentrained mPFC cells selectively encoded a variety of task relevant behaviors and stimuli while never theta-entrained mPFC cells were most strongly attuned to errors or the lack of expected

Spatial mapping and navigation are figured prominently in the extant literature that describes hippocampal function. The medial entorhinal cortex is likewise attracting increasing interest, insofar as evidence accumulates that this area also contributes to spatial information processing. Here, we discuss recent electrophysiological findings that offer an alternate view of hippocampal and medial entorhinal function. These findings suggest complementary contributions of the hippocampus and medial entorhinal cortex in support of episodic memory, wherein hippocampal networks encode sequences of events that compose temporally and spatially extended episodes, whereas medial entorhinal networks disambiguate overlapping episodes by binding sequential events into distinct memories. Copyright © 2008 P. A. Lipton and H. Eichenbaum. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. 1. THE BRAIN’S GPS Does hippocampal activity embody the cognitive map? One should expect the neural instantiation of Tolman’s [1] cognitive map to contain units (neurons) that are fully allocentric, that is, identify places in the environment independent of