Humans can be instructed verbally to perform computationally complex cognitive tasks; their performance then improves relatively slowly over the course of practice. Many skills underlie these abilities; in this paper, we focus on the particular question of a uniform architecture for the instantiation of habitual performance and the storage, recall, and execution of simple rules. Our account builds on models of gated working memory, and involves a bilinear architecture for representing conditional input-output maps and for matching rules to the state of the input and working memory. We demonstrate the performance of our model on two paradigmatic tasks used to investigate prefrontal and basal ganglia function.

Abstract not found

Although we often encounter circumstances with which we have no prior experience, we rapidly learn how to behave in these novel situations. Such adaptive behavior relies on abstract behavioral rules that are generalizable, rather than concrete rules mapping specific cues to specific responses. Although the frontal cortex is known to support concrete rule learning, less well understood are the neural mechanisms supporting the acquisition of abstract rules. Here, we use a reinforcement learning paradigm to demonstrate that more anterior regions along the rostro-caudal axis of frontal cortex support rule learning at higher levels of abstraction. Moreover, these results indicate that when humans confront new rule learning problems, this rostro-caudal division of labor supports the search for relationships between context and action at multiple levels of abstraction simultaneously.

■ Appetitive goal-directed behavior can be associated with a cue-triggered expectancy that it will lead to a particular reward, a process thought to depend on the OFC and basolateral amygdala complex. We developed a biologically informed neural network model of this system to investigate the separable and complementary roles of these areas as the main components of a flexible expectancy system. These areas of interest are part of a neural network with additional subcortical areas, including the central nucleus of amygdala, ventral (limbic) and dorsomedial (associative) striatum. Our simulations are consistent with the view that the amygdala maintains Pavlovian associations through incremental updating of synaptic strength and that the OFC supports flexibility by maintaining an activation-based working memory of the recent reward history. Our model provides a mechanistic explanation for electrophysiological evidence that cue-related firing in OFC neurons is nonselectively early after a contingency change and why this nonselective firing is critical for promoting plasticity in the amygdala. This ambiguous activation results from the simultaneous maintenance of recent outcomes and obsolete Pavlovian contingencies in working memory. Furthermore, at the beginning of reversal, the OFC is critical for supporting responses that are no longer inappropriate. This result is inconsistent with an exclusive inhibitory account of OFC function. ■

This paper presents an embodied biologically-plausible model investigating the relationships existing between Pavlovian and instrumental conditioning. The model is validated by successfully reproducing the primary outcomes of instrumental-conditioning devaluation tests conducted with normal and amygdala-lesioned rats. These experiments are particularly important as they show how the sensitivity to motivational states exhibited by the Pavlovian system can transfer to instrumentally acquired behaviors. The results presented are relevant not only for neuroscience but also for robotics as they start to investigate how internal motivational systems, as those found in real organisms, might modulate the learning and performance of goal-directed actions in artificial machines, so to improve their behavioral flexibility. 1

Current models of reinforcement learning are based on the assumption that learning must be guided by rewarding (unconditioned) stimuli. On the other hand, there is empirical evidence that dopamine bursts, which are commonly considered as the reinforcement learning signals, can also be triggered by apparently neutral stimuli, and that this can lead to conditioning phenomena in absence of any rewarding stimuli. In this paper we present a computational model, based on an hypothesis proposed in Redgrave and Gurney (2006), in which dopamine release is directly triggered by the superior colliculus (a dorsal midbrain structure) when it detects novel visual stimuli and this supports instrumental conditioning. The model incorporates various biological constraints, for example the anatomical and physiological data related to the micro-architecture of the superior colliculus presented in Binns and Salt (1997). The model is validated by reproducing with a simulated robotic rat the results of an experiment with real rats on the role of intrinsically reinforcing properties of apparently neutral stimuli reported in Reed et al. (1996). 1

■ A paradigmatic test of executive control, the n-back task, is known to recruit a widely distributed parietal, frontal, and striatal “executive network, ” and is thought to require an equally wide array of executive functions. The mapping of functions onto substrates in such a complex task presents a significant challenge to any theoretical framework for executive control. To address this challenge, we developed a biologically constrained model of the n-back task that emergently develops the ability to appropriately gate, bind, and maintain information in working memory in the course of learning to perform the task. Furthermore, the model is sensitive to proactive interference in ways that match findings from neuroimaging and shows a U-shaped performance curve after manipulation of prefrontal dopaminergic mechanisms similar to that observed in studies of genetic polymorphisms and pharmacological manipulations. Our model represents a formal computational link between anatomical, functional neuroimaging, genetic, behavioral, and theoretical levels of analysis in the study of executive control. In addition, the model specifies one way in which the pFC, BG, parietal, and sensory cortices may learn to cooperate and give rise to executive control. ■

doi: 10.3389/fnins.2012.00018 Acetylcholine-based entropy in response selection: a model of how striatal interneurons modulate exploration, exploitation, and response variability in decision-making

Humans can be instructed verbally to perform computationally complex cognitive tasks; their performance then improves relatively slowly over the course of practice. Many skills underlie these abilities; in this paper, we focus on the particular question of a uniform architecture for the instantiation of habitual performance and the storage, recall, and execution of simple rules. Our account builds on models of gated working memory, and involves a bilinear architecture for representing conditional input-output maps and for matching rules to the state of the input and working memory. We demonstrate the performance of our model on two paradigmatic tasks used to investigate prefrontal and basal ganglia function.