A large number of computational models of information processing in the basal ganglia have been developed in recent years. Prominent in these are actor–critic models of basal ganglia functioning, which build on the strong resemblance between dopamine neuron activity and the temporal difference prediction error signal in the critic, and between dopamine-dependent long-term synaptic plasticity in the striatum and learning guided by a prediction error signal in the actor. We selectively review several actor–critic models of the basal ganglia with an emphasis on two important aspects: the way in which models of the critic reproduce the temporal dynamics of dopamine firing, and the extent to which models of the actor take into account known basal ganglia anatomy and physiology. To complement the efforts to relate basal ganglia mechanisms to reinforcement learning (RL), we introduce an alternative approach to modeling a critic network, which uses Evolutionary Computation techniques to ‘evolve ’ an optimal RL mechanism, and relate the evolved mechanism to the basic model of the critic. We conclude our discussion of models of the critic by a critical discussion of the anatomical plausibility of implementations of a critic in basal ganglia circuitry, and conclude that such implementations build on assumptions that are inconsistent with the known anatomy of the basal ganglia. We return to the actor component of the actor–critic model, which is usually modeled at the striatal level with very little detail. We describe an alternative model of the basal ganglia which takes into account several important, and previously neglected, anatomical and physiological characteristics of basal ganglia–thalamocortical connectivity and suggests that the basal ganglia performs reinforcement biased dimensionality reduction of cortical inputs. We further suggest that since such selective encoding may bias the representation at the

The concept of error detection plays a central role in theories of executive control. In this article, the authors present a mechanism that can rapidly detect errors in speeded response time tasks. This error monitor assigns values to the output of cognitive processes involved in stimulus categorization and response generation and detects errors by identifying states of the system associated with negative value. The mechanism is formalized in a computational model based on a recent theoretical framework for understanding error processing in humans (C. B. Holroyd & M. G. H. Coles, 2002). The model is used to simulate behavioral and event-related brain potential data in a speeded response time task, and the results of the simulation are compared with empirical data. Frontal parts of the brain, including the prefrontal cortex (Luria, 1973; Stuss & Knight, 2002), the anterior cingulate cortex (Devinsky, Morrell, & Vogt, 1995; Posner & DiGirolamo, 1998), and their connections with the basal ganglia (L. L. Brown, Schneider, & Lidsky, 1997; Cummings, 1993), are thought to compose an executive system for cognitive control. The functions of this system are thought to include setting high-level goals, directing other

The primate cortex represents perceived and produced events in a distributed way, which calls for a mechanism that integrates their features into coherent structures. Animal, drug, and patient studies suggest that the local binding of visual features is under muscarinic–cholinergic control, whereas visuomotor binding seems to be driven by dopaminergic pathways. Consistent with this picture, we present evidence that the binding of visual features and actions is modulated by stress, induced by the cold pressure test (CPT), which causes an excessive dopamine turnover in prefrontal cortex. The impact of stress was restricted to the task-relevant visuomotor binding, supporting claims that dopamine affects the maintenance of task-relevant information in working memory. The outcome pattern, including the impact of the personality trait extraversion, suggests that the relation between dopamine level and visuomotor performance follows an inverted U-shaped function, with strongest binding being associated with average dopamine levels.

The primate cortex represents the external world in a distributed fashion, which calls for a mechanism that integrates and binds the features of a perceived or processed event. Animal and patients studies provide evidence that feature binding in the visual cortex is driven by the muscarinic–cholinergic system, whereas visuo-motor integration may be under dopaminergic control. Consistent with this scenario, we present indication that the binding of visual and action features is modulated by emotions through the probable stimulation of the dopaminergic system. Interestingly, the impact of emotions on binding was restricted to tasks in which shape was task-relevant, suggesting that extracting affective information is not automatic but requires attention to shape. © 2006 Elsevier Ltd. All rights reserved.

Abstract not found

Abstract not found

FOREWORD This thesis was carried out in close collaboration with my supervisor Neslihan Serap ¸Sengör over the last two and a half years. I would like to thank her for introducing me the field, giving advices, sharing her opinions and listening to mine. It is certain that this work will not be possible without her generous support. I would like to thank Fatih ˙I¸sbecer, my manager at Pozitron, for his unfailing support which has enabled me to complete this work. In the first year of this research, I have attended to Behavioral Neurology course given by Hakan Gürvit at Istanbul University. I would like to thank him for giving me this opportunity and kindly supporting anyone interested in the subject. It was a great pleasure for me to meet Yuichi Sakumura and Ricardo Carnieri at Okinawa Computational Neuroscience Course in 2006. I would like to thank Yuichi Sakumura for his ongoing support and friendship. I would also like to thank Ricardo Carnieri for his friendship and the technical talks we had during the hours we spent on the project work. Finally, I would like to dedicate this thesis to my family. They are going to be happier than me seeing this work finished. May 2007 Mete Balcı iii CONTENTS FOREWORD iii

Volition, like intelligence, is a concept of interest and utility to both philosophers and researchers in artificial intelligence. Unfortunately, it is often poorly defined, potentially difficult to assess in biological and artificial systems, and its usage recalls the ancient, futile debate of free will vs. determinism. This paper proposes to define volition, and to suggest a functionally-defined, physically-grounded ordinal scale and a procedure by which volition might be measured: a kind of Turing test for volition, but motivated by an explicit analysis of the concept being tested and providing results which are graded, rather than Boolean, so that candidate systems may be ranked according to their degree of volitional endowment. Volition is proposed to be a functional, aggregate property of certain physical systems and is defined as the capacity for adaptive decision-making. A scale similar in scope to Daniel Dennett’s Kinds of Minds scale is then proposed, as well as a set of progressive “litmus tests ” for determining where a candidate system falls on the scale (see Tables 1-4). Such a scale could be useful in illuminating our understanding of volition and in assessing progress made in engineering intelligent, autonomous artificial organisms.

Adapting decision-making according to dynamic and probabilistic changes in action-reward contingencies is critical for survival in a competitive and resource-limited world. Much research has focused on elucidating the neural systems and computations that underlie how the brain identifies whether the consequences of actions are relatively good or bad. In contrast, less empirical research has focused on the mechanisms by which reinforcements might be used to guide decision-making. Here I review recent studies that have attempted to bridge this gap by characterizing how humans use reward information to guide and optimize decision-making. Regions that have been implicated in reinforcement processing, including the striatum, orbitofrontal cortex, and anterior cingulate, also seem to mediate how reinforcements are used to adjust subsequent decision-making. This research provides insights into why the brain devotes resources to evaluating reinforcements, and suggests a direction for future research, from studying the mechanisms of reinforcement processing to the mechanisms of reinforcement learning.

of cocaine sensitization on dorsolateral and ventral striatum in the context of an Actor/Critic model