The prefrontal cortex has long been thought to subserve both working memory (the holding of information online for processing) and executive functions (deciding how to manipulate working memory and perform processing). Although many computational models of working memory have been developed, the mechanistic basis of executive function remains elusive, often amounting to a homunculus. This article presents an attempt to deconstruct this homunculus through powerful learning mechanisms that allow a computational model of the prefrontal cortex to control both itself and other brain areas in a strategic, task-appropriate manner. These learning mechanisms are based on subcortical structures in the midbrain, basal ganglia, and amygdala, which together form an actor-critic architecture. The critic system learns which prefrontal representations are task relevant and trains the actor, which in turn provides a dynamic gating mechanism for controlling working memory updating. Computationally, the learning mechanism is designed to simultaneously solve the temporal and structural credit assignment problems. The model’s performance compares favorably with standard backpropagation-based temporal learning mechanisms on the challenging 1-2-AX working memory task and other benchmark working memory tasks.

The frontal cortex and basal ganglia interact via a relatively well-understood and elaborate system of interconnections. In the context of motor function, these interconnections can be understood as disinhibiting or "releasing the brakes" on frontal motor action plans --- the basal ganglia detect appropriate contexts for performing motor actions, and enable the frontal cortex to execute such actions at the appropriate time. We build on this idea in the domain of working memory through the use of computational neural network models of this circuit. In our model, the frontal cortex exhibits robust active maintenance, while the basal ganglia contribute a selective, dynamic gating function that enables frontal memory representations to be rapidly updated in a task-relevant manner. We apply the model to a novel version of the continuous performance task (CPT) that requires subroutine-like selective working memory updating, and compare and contrast our model with other existing models and th...

This review proposes that implicit learning processes are the cognitive substrate of social intuition. This hypothesis is supported by (a) the conceptual correspondence between implicit learning and social intuition (nonverbal communication) and (b) a review of relevant neuropsychological (Huntington's and Parkinson's disease), neuroimaging, neurophysiological, and neuroanatomical data. It is concluded that the caudate and putamen, in the basal ganglia, are central components of both intuition and implicit learning, supporting the proposed relationship. Parallel, but distinct, processes of judgment and action are demonstrated at each of the social, cognitive, and neural levels of analysis. Additionally, explicit attempts to learn a sequence can interfere with implicit learning. The possible relevance of the computations of the basal ganglia to emotional appraisal, automatic evaluation, script processing, and decision making are discussed. These "feelings " have an efficiency of operation which it is impossi-ble for thought to match. Even our most highly intellectualized operations depend upon them as a "fringe " by which to guide our inferential movements. They give us our sense of rightness and wrongness, of what to select and emphasize and follow up, and what

The authors propose a reinforcement-learning mechanism as a model for recurrent choice and extend it to account for skill learning. The model was inspired by recent research in neurophysiological studies of the basal ganglia and provides an integrated explanation of recurrent choice behavior and skill learning. The behavior includes effects of differential probabilities, magnitudes, variabilities, and delay of reinforcement. The model can also produce the violation of independence, preference reversals, and the goal gradient of reinforcement in maze learning. An experiment was conducted to study learning of action sequences in a multistep task. The fit of the model to the data demonstrated its ability to account for complex skill learning. The advantages of incorporating the mechanism into a larger cognitive architecture are discussed.

Is attention a purely perceptual process or is it in any way related to motor control? The aim of this article is to show that attention puts similar demands on a cognitive system as motor control and present evidence supporting the view that similar mechanisms operate in the two processes. A computational model of attention is presented that uses habituation as well as classical and instrumental conditioning to explain a number of attentional processes. Evidence from neurophysiology is reviewed that suggest that attention is controlled in a way similar to actions. This view makes it possible to adapt traditional learning theoretical mechanisms to the control of attention. Computer simulations are presented that illustrates the operation of the model.

Instrumental (or operant) conditioning, a form of animal learning, is similar to reinforcement learning (Watkins, 1989) in that it allows an agent to adapt its actions to gain maximally from the environment while only being rewarded for correct performance. But animals learn much more complicated behaviors through instrumental conditioning than robots presently acquire through reinforcement learning. We describe a new computational model of the conditioning process that attempts to capture some of the aspects that are missing from simple reinforcement learning: conditioned reinforcers, shifting reinforcement contingencies, explicit action sequencing, and state space re nement. We apply our model to a task commonly used to study working memory in rats and monkeys: the DMTS (Delayed Match to Sample) task. Animals learn this task in stages. In simulation, our model also acquires the task in stages, in a similar manner. We have used the model to train an RWI B21 robot.

Foraging- and feeding-related behaviors across eumetazoans share similar molecular mechanisms, suggesting the early evolution of an optimal foraging behavior called area-restricted search (ARS), involving mechanisms of dopamine and glutamate in the modulation of behavioral focus. Similar mechanisms in the vertebrate basal ganglia control motor behavior and cognition and reveal an evolutionary progression toward increasing internal connections between prefrontal cortex and striatum in moving from amphibian to primate. The basal ganglia in higher vertebrates show the ability to transfer dopaminergic activity from unconditioned stimuli to conditioned stimuli. The evolutionary role of dopamine in the modulation of goal-directed behavior and cognition is further supported by pathologies of human goal-directed cognition, which have motor and cognitive dysfunction and organize themselves, with respect to dopaminergic activity, along the gradient described by ARS, from perseverative to unfocused. The evidence strongly supports the evolution of goal-directed cognition out of mechanisms initially in control of spatial foraging but, through increasing cortical connections, eventually used to forage for information.

The ability to predict future events based on the past is an important attribute of organisms that engage in adaptive behaviour. One prominent computational method for learning to predict is called temporal-difference (TD) learning. It is so named because it uses the difference between successive predictions to learn to predict correctly. TD learning is well suited to modelling the biological phenomenon of conditioning, wherein an organism learns to predict a reward even though the reward may occur later in time. We review a model for conditioning in bees based on TD learning. The model illustrates how the TD-learning algorithm allows an organism to learn an appropriate sequence of actions leading up to a reward, based solely on reinforcement signals. The second part of the paper describes how TD learning can be used at the cellular level to model the recently discovered phenomenon of spike-timing-dependent plasticity. Using a biophysical model of a neocortical neuron, we demonstrate that the shape of the spike-timing-dependent learning windows found in biology can be interpreted as a form of TD learning occurring at the cellular level. We conclude by showing that such spike-based TD-learning mechanisms can produce direction selectivity in visual-motion-sensitive cells and can endow recurrent neocortical circuits with the powerful ability to predict their inputs at the millisecond time-scale.

The basal ganglia (BG) have been hypothesized to perform reinforcement learning by use of reinforcement signals provided by dopamine neurons. It is well known that there exist multiple BG-thalamocortical loops, but their functions are poorly understood. Here, we propose a computational model of how different BG loops are employed in visuomotor sequence learning using different representations of sequence. The central idea of the model is that a visuomotor sequence is easier to learn in spatial representation (e.g. visual coordinates) but is easier to control in bodybased representation (e.g. joint angle coordinates). The results of simulations of the model replicated both behavioral and neurophysiological findings in recent experimental studies using "2x5 task". Keywords--- reinforcement learning, basal ganglia, procedural memory, visuomotor learning I. INTRODUCTION Schultz and his colleagues has shown in their experiments [1] that the response tuning of dopamine (DA) neurons in the ...

The existing procedural learning mechanism in ACT-R (Anderson & Lebiere, 1998) has been successful in explaining a wide range of adaptive choice behavior. However, the existing mechanism is inherently limited to learning from binary feedback (i.e. whether a reward is received or not). It is thus difficult to capture choice behavior that is sensitive to both the probabilities of receiving a reward and the reward magnitudes. By modifying the temporal difference learning algorithm (Sutton & Barto, 1998), a new procedural learning mechanism is implemented that generalizes and extends the computational abilities of the current mechanism. Models using the new mechanism were fit to three sets of human data collected from experiments of probability learning and decision making tasks. The new procedural learning mechanism fit the data at least as well as the existing mechanism, and is able to fit data that are problematic for the existing mechanism. This paper also shows how the principle of reinforcement learning can be implemented in a production system like ACT-R.