Intrinsic motivation is a crucial mechanism for open-ended cognitive development since it is the driver of spontaneous exploration and curiosity. Yet, it has so far only been conceptualized in ad hoc manners in the epigenetic robotics community. After reviewing different approaches to intrinsic motivation in psychology, this paper presents a unified definition of intrinsic motivation, based on the theory of Daniel Berlyne. Based on this definition, we propose a landscape of types of computational approaches, making it possible to position existing and future models relative to each other, and we show that important approaches are still to be explored. 1.

There is great interest in building intrinsic motivation into artificial systems using the reinforcement learning framework. Yet, what intrinsic motivation may mean computationally, and how it may differ from extrinsic motivation, remains a murky and controversial subject. In this article, we adopt an evolutionary perspective and define a new optimal reward framework that captures the pressure to design good primary reward functions that lead to evolutionary success across environments. The results of two computational experiments show that optimal primary reward signals may yield both emergent intrinsic and extrinsic motivation. The evolutionary perspective and the associated optimal reward framework thus lead to the conclusion that there are no hard and fast features distinguishing intrinsic and extrinsic reward computationally. Rather, the directness of the relationship between rewarding behavior and evolutionary success varies along a continuum.

Reinforcement learning has achieved broad and successful application in cognitive science in part because of its general formulation of the adaptive control problem as the maximization of a scalar reward function. The computational reinforcement learning framework is motivated by correspondences to animal reward processes, but it leaves the source and nature of the rewards unspecified. This paper advances a general computational framework for reward that places it in an evolutionary context, formulating a notion of an optimal reward function given a fitness function and some distribution of environments. Novel results from computational experiments show how traditional notions of extrinsically and intrinsically motivated behaviors may emerge from such optimal reward functions. In the experiments these rewards are discovered through automated search rather than crafted by hand. The precise form of the optimal reward functions need not bear a direct relationship to the fitness function, but may nonetheless confer significant advantages over rewards based only on fitness.

Abstract—One important role of an agent’s motivational system is to choose, at any given moment, which of a number of skills the agent should attempt to improve. Many researchers have suggested “intrinsically motivated ” systems that receive internal reward for model learning progress, but for the most part this notion has not been applied with respect to skill competence, or to choose between skills. In this paper we propose an agent motivated to gain competence in its environment by learning a number of skills, addressing head-on the mechanism of competence progress motivation for the purpose of governing the efficient learning of skills. We demonstrate this new approach in a simple illustrative domain and show that it outperforms a naïve agent, achieving higher competence faster by focusing attention and learning effort on skills for which progress can be made while ignoring those skills that are already learned or are at the moment too difficult. I.

Recent advances in robotics leads us to consider, on the one hand, the notion of a kernel, a set of stable algorithms that drive developmental dynamics and, on the other hand, variable body envelopes that change over time. This division reverses the classic notion of a fixed body on which different software can be applied to consider a fixed software that can be applied to different kinds of embodiment. Thus, it becomes possible to study how a particular embodiment shapes developmental trajectories in specific ways. It also leads us to a novel view of the development of skills, from sensorimotor dexterity to abstract thought, based on the notion of a fluid body in continuous redefinition. 1

We describe a robot system that autonomously acquires skills through interaction with its environment. The robot learns to sequence the execution of a set of innate controllers to solve a task, extracts and retains components of that solution as portable skills, and then transfers those skills to reduce the time required to learn to solve a second task.

Children seem to acquire new know-how in a continuous and open-ended manner. In this paper, we hypothesize that an intrinsic motivation to progress in learning is at the origins of the remarkable structure of children’s developmental trajectories. In this view, children engage in exploratory and playful activities for their own sake, not as steps toward other extrinsic goals. The central hypothesis of this paper is that intrinsically motivating activities correspond to expected decrease in prediction error. This motivation system pushes the infant to avoid both predictable and unpredictable situations in order to focus on the ones that are expected to maximize progress in learning. Based on a computational model and a series of robotic experiments, we show how this principle can lead to organized sequences of behavior of increasing complexity characteristic of several behavioral and developmental patterns observed in humans. We then discuss the putative circuitry underlying such an intrinsic motivation system in the brain and formulate two novel hypotheses. The first one is that tonic dopamine acts as a learning progress signal. The second is that this progress signal is directly computed through a hierarchy of microcortical circuits that act both as prediction and metaprediction systems.

Language acquisition has probably been the centre of the most profound, and yet unsolved, scientific debates in cognitive sciences in the 20 th century. Computational and robotic models elaborated in the recent years aim at overcoming a number of conceptual barriers in this debate, because they help to naturalize these questions and allow us for the first time to confront theories to reality. This issue of the newsletter features a dialog, initiated by Angelo Cangelosi, over the symbol grounding problem, in relation to advances in robotics, machine learning and artificial intelligence. Central scientific actors of the history of this problem have responded. Stevan Harnad, Luc Steels, Aaron Sloman, Stephen Cowley, Vincent Müller, Carol Madden, Peter Ford Dominey and Stéphane Lallée present their own, sometimes very constrastive, views, and we see that in spite of some clear progress, a lot of work is in front of us, both conceptually and in terms of robotic experiments. Then, a novel call for dialog is proposed by Gianluca Baldassarre and Marco Mirolli and relates to the challenges of cumulative learning: « What are the key open challenges for understanding autonomous cumulative learning of skills? ». Interested researchers are welcome to submit a response (contact pierre-yves.oudeyer@inria.fr) by September 1 st, 2010. The length of each response must be between 300 and 500 words (including references). I take the opportunity of this editorial to congratulate Zhengyou Zhang for his work as a chair of the AMD TC, which has been essential for the growth of our community, and in particular his strong support for this newsletter. I also welcome Minoru Asada as the new AMD TC chair, and I am sure his vision will help all of us to increase even more the momentum of computational developmental sciences.

Intrinsic motivation is a crucial mechanism for open-ended cognitive development since it is the driver of spontaneous exploration and curiosity. Yet, it has so far only been conceptualized in ad hoc manners in the epigenetic robotics community. After reviewing different approaches to intrinsic motivation in psychology, this paper presents a unified definition of intrinsic motivation, based on the theory of Daniel Berlyne. Based on this definition, we propose a landscape of types of computational approaches, making it possible to position existing and future models relative to each other, and we show that important approaches are still to be explored. 1.

c t i v it y e p o r t