Abstract—Exploratory activities seem to be intrinsically rewarding for children and crucial for their cognitive development. Can a machine be endowed with such an intrinsic motivation system? This is the question we study in this paper, presenting a number of computational systems that try to capture this drive towards novel or curious situations. After discussing related research coming from developmental psychology, neuroscience, developmental robotics, and active learning, this paper presents the mechanism of Intelligent Adaptive Curiosity, an intrinsic motivation system which pushes a robot towards situations in which it maximizes its learning progress. This drive makes the robot focus on situations which are neither too predictable nor too unpredictable, thus permitting autonomous mental development. The complexity of the robot’s activities autonomously increases and complex developmental sequences self-organize without

Ever since William James, psychologists of emotion have tended to view affective states as intrinsically conscious. We argue that nonconscious affect also exists, and focus specifically on the possibility of unconscious ``liking''. We present evidence that positive and negative affective reactions can be elicited subliminally, while a person is completely unaware of any affective reaction at all �in addition to being unaware of the causal stimulus). Despite the absence of any detectable subjective experience of emotion, subliminally induced unconscious ``liking' ' can influence later consumption behaviour. We suggest that unconscious ``liking' ' is mediated by specific subcortical brain systems, such as the nucleus accumbens and its connections. Ordinarily, conscious liking �feelings of pleasure) results from the interaction of separate brain systems of conscious awareness with those core processes of unconscious affect. But under some conditions, activity in brain systems mediating unconscious core ``liking' ' may become decoupled from conscious awareness. The result is a genuinely unconscious emotion. We begin with apologies to William James for having stolen the title of our paper from his classic article, ``What is an emotion' ' �James, 1884). Worse still, by inserting ``unconscious' ' as a modifier, our title distorts his concept of emotion in a way that renders it almost nonsensical. This is because an unconscious emotion was a contradiction in terms, according to James ' �1884) definition. For James, emotion was a conscious experience or subjective feeling

This article addresses the relationship between long-term reward predictions and slow-timescale neural activity in temporal difference (TD) models of the dopamine system. Such models attempt to explain how the activity of dopamine (DA) neurons relates to errors in the prediction of future rewards. Previous models have been mostly restricted to short-term predictions of rewards expected during a single, somewhat artificially defined trial. Also, the models focused exclusively on the phasic pause-and-burst activity of primate DA neurons; the neurons' slower, tonic background activity was assumed to be constant. This has led to difficulty in explaining the results of neurochemical experiments that measure indications of DA release on a slow timescale, results that seem at first glance inconsistent with a reward prediction model. In this article, we investigate a TD model of DA activity modified so as to enable it to make longer-term predictions about rewards expected far in the future. We show that these predictions manifest themselves as slow changes in the baseline error signal, which we associate with tonic DA activity. Using this model, we make new predictions about the behavior of the DA system in a number of experimental situations. Some of these predictions suggest new computational explanations for previously puzzling data, such as indications from microdialysis studies of elevated DA activity triggered by aversive events

Abstract: A wealth of research focuses on the decision-making processes that animals and humans employ when selecting actions in the face of reward and punishment. Initially such work stemmed from psychological investigations of conditioned behavior, and explanations of these in terms of computational models. Increasingly, analysis at the computational level has drawn on ideas from reinforcement learning, which provide a normative framework within which decision-making can be analyzed. More recently, the fruits of these extensive lines of research have made contact with investigations into the neural basis of decision making. Converging evidence now links reinforcement learning to specific neural substrates, assigning them precise computational roles. Specifically, electrophysiological recordings in behaving animals and functional imaging of human decision-making have revealed in the brain the existence of a key reinforcement learning signal, the temporal difference reward prediction error. Here, we first introduce the formal reinforcement learning framework. We then review the multiple lines of evidence linking reinforcement learning to the function of dopaminergic neurons in the mammalian midbrain and

This dissertation would not have been possible without the help and support of many people. Most of all, I would like to extend my gratitude to Rod Grupen for many years of inspiring work, our discussions, and his guidance. Without his support and vision, I cannot imagine that the journey would have been as enormously enjoyable and rewarding as it turned out to be. I am very excited about what we discovered during my time at UMass, but there is much more to be done. I look forward to what comes next! In addition to providing professional inspiration, Rod was a great person to work with and for—creating a warm and encouraging laboratory atmosphere, motivating us to stay in shape for his annual half-marathons, and ensuring a sufficient amount of cake at the weekly lab meetings. Thanks for all your support, Rod! I am very grateful to my thesis committee—Andy Barto, David Jensen, and Rachel Keen—for many encouraging and inspirational discussions. Their comments and feedback significantly contributed to the form of this document. I would especially

Abstract. Children seem intrinsically motivated to manipulate, to explore, to test, to learn and they look for activities and situations that provide such learning opportunities. Inspired by research in developmental psychology and neuroscience, some researchers have started to address the problem of designing intrinsic motivation systems. A robot controlled by such systems is able to autonomously explore its environment not to fulfil predefined tasks but driven by an incentive to search for situations where learning happens efficiently. In this paper, we present the origins of these intrinsically motivated machines, our own research in this novel field and we argue that intrinsic motivation might be a crucial step towards machines capable of life-long learning and open-ended development.

Children seem to acquire new know-how in a continuous and open-ended manner. In this paper, we hypothesize that an intrinsic motivation to progress in learning is at the origins of the remarkable structure of children’s developmental trajectories. In this view, children engage in exploratory and playful activities for their own sake, not as steps toward other extrinsic goals. The central hypothesis of this paper is that intrinsically motivating activities correspond to expected decrease in prediction error. This motivation system pushes the infant to avoid both predictable and unpredictable situations in order to focus on the ones that are expected to maximize progress in learning. Based on a computational model and a series of robotic experiments, we show how this principle can lead to organized sequences of behavior of increasing complexity characteristic of several behavioral and developmental patterns observed in humans. We then discuss the putative circuitry underlying such an intrinsic motivation system in the brain and formulate two novel hypotheses. The first one is that tonic dopamine acts as a learning progress signal. The second is that this progress signal is directly computed through a hierarchy of microcortical circuits that act both as prediction and metaprediction systems.

The desire to seek new and unfamiliar experiences is a fundamental behavioral tendency in humans and other species. In economic decision making, novelty seeking is often rational, insofar as uncertain options may prove valuable and advantageous in the long run. Here, we show that, even when the degree of perceptual familiarity of an option is unrelated to choice outcome, novelty nevertheless drives choice behavior. Using functional magnetic resonance imaging (fMRI), we show that this behavior is specifically associated with striatal activity, in a manner consistent with computational accounts of decision making under uncertainty. Furthermore, this activity predicts interindividual differences in susceptibility to novelty. These data indicate that the brain uses perceptual novelty to approximate choice uncertainty in decision making, which in certain contexts gives rise to a newly identified and quantifiable source of human irrationality.

Abstract not found

Recent advances in the cognitive neuroscience of motivation and learning have demonstrated a critical role for midbrain dopamine and its targets in reward prediction. Converging evidence suggests that midbrain dopamine neurons signal a reward prediction error, allowing an organism to predict, and to act to increase, the probability of reward in the future. This view has been highly successful in accounting for a wide range of reinforcement learning phenomena in animals and humans. However, while current theories of midbrain dopamine provide a good account of behavior known as habitual or stimulus-response learning, we review evidence suggesting that other neural and cognitive processes are involved in motivated, goal-directed behavior. We discuss how this distinction resembles the classic distinction in the cognitive neuroscience of memory between nondeclarative and declarative memory systems, and discuss common themes between mnemonic and motivational functions. Finally, we present data demonstrating links between mnemonic processes and reinforcement learning. The past decade has seen a growth of interest in the cognitive neuroscience of motivation and reward. This is largely rooted in a series of neurophysiology studies of the response properties of dopamine-containing midbrain neurons in primates receiving reward (Schultz, 1998). The responses of these neurons were subsequently interpreted in terms of reinforcement learning, a computational framework for trial and error learning from reward (Houk, Adams, & Barto, 1995; Montague, Dayan, & Sejnowski, 1996; Schultz, Dayan, & Montague, 1997). Together with Both authors contributed equally to this article. We are most grateful to Shanti Shanker for assistance with data collection, to Anthony Wagner for generously allowing us to conduct the experiment reported here in his laboratory, and to Alison Adcock, Lila Davachi, Peter Dayan, Mark