Abstract — Robots that work alongside us in our homes and workplaces could extend the time an elderly person can live at home, provide physical assistance to a worker on an assembly line, or help with household chores. In order to assist us in these ways, robots will need to successfully perform manipulation tasks within human environments. Human environments present special challenges for robot manipulation since they are complex, dynamic, uncontrolled, and difficult to perceive reliably. In this paper we present a behavior-based control system that enables a humanoid robot, Domo, to help a person place objects on a shelf. Domo is able to physically locate the shelf, socially cue a person to hand it an object, grasp the object that has been handed to it, transfer the object to the hand that is closest to the shelf, and place the object on the shelf. We use this behavior-based control system to illustrate three themes that characterize our approach to manipulation in human environments. The first theme, cooperative manipulation, refers to the advantages that can be gained by having the robot work with a person to cooperatively perform manipulation tasks. The second theme, task relevant features, emphasizes the benefits of carefully selecting the aspects of the world that are to be perceived and acted upon during a manipulation task. The third theme, let the body do the thinking, encompasses several ways in which a robot can use its body to simplify manipulation tasks. 1 Fig. 1. The humanoid robot Domo used in this paper. I.

Abstract — Estimating the traversability of terrain in an unstructured outdoor environment is a core functionality for autonomous robot navigation. While general-purpose sensing can be used to identify the existence of terrain features such as vegetation and sloping ground, the traversability of these regions is a complex function of the terrain characteristics and vehicle capabilities, which makes it extremely difficult to characterize a priori. Moreover, it is difficult to find general rules which work for a wide variety of terrain types such as trees, rocks, tall grass, logs, and bushes. As a result, methods which provide traversability estimates based on predefined terrain properties such as height or shape will be unlikely to work reliably in unknown outdoor environments. Our approach is based on the observation that traversability in the most general sense is an affordance which is jointly determined by the vehicle and its environment. We describe a novel on-line learning method which can make accurate predictions of the traversability properties of complex terrain. Our method is based on autonomous training data collection which exploits the robot’s experience in navigating its environment to train classifiers without human intervention. This is in contrast to other learning methods in which training data is collected manually. We have implemented and tested our traversability learning method on an ummaned ground vehicle (UGV) and evaluated its performance in several realistic outdoor environments. The experiments quantify the benefit of our on-line traversability learning approach. I.

Abstract — This work is concerned with a framework for visual object recognition in real world tasks. Our approach is motivated by biological findings of the representation of space around the body, the so-called peripersonal space. We show that the principles behind those findings can lead to a natural structuring of object recognition tasks in artificial systems. We demonstrate this by the supervised learning and recognition of 20 complexshaped objects from unsegmented visual input.

Abstract — We present a developmental perceptual system for a humanoid robot that autonomously discovers its hand from less than 2 minutes of natural interaction with a human. The perceptual system combines simple proprioceptive sensing with a visual attention system that uses motion to select salient regions. We show that during natural interactions with a person, the majority of the selected visual regions consist of significant body parts on the human and robot (hands, fingers, and the human’s head). The system visually clusters the selected image regions, models their spatial distribution over a sensory sphere, and uses mutual information to determine how much the clusters are influenced by the robot’s arm. In our tests, the visual cluster that most strongly relates to the robot’s arm primarily contains images of the robot’s hand, and has a spatial distribution that can predict the location of the robot’s hand in the image as a function of the arm’s configuration. 1 I.

Affordances encode relationships between actions, objects, and effects. They play an important role on basic cognitive capabilities such as prediction and planning. We address the problem of learning affordances through the interaction of a robot with the environment, a key step to understand the world properties and develop social skills. We present a general model for learning object affordances using Bayesian networks integrated within a general developmental architecture for social robots. Since learning is based on a probabilistic model, the approach is able to deal with uncertainty, redundancy, and irrelevant information. We demonstrate successful learning in the real world by having an humanoid robot interacting with objects. We illustrate the benefits of the acquired knowledge in imitation games.

This dissertation would not have been possible without the help and support of many people. Most of all, I would like to extend my gratitude to Rod Grupen for many years of inspiring work, our discussions, and his guidance. Without his support and vision, I cannot imagine that the journey would have been as enormously enjoyable and rewarding as it turned out to be. I am very excited about what we discovered during my time at UMass, but there is much more to be done. I look forward to what comes next! In addition to providing professional inspiration, Rod was a great person to work with and for—creating a warm and encouraging laboratory atmosphere, motivating us to stay in shape for his annual half-marathons, and ensuring a sufficient amount of cake at the weekly lab meetings. Thanks for all your support, Rod! I am very grateful to my thesis committee—Andy Barto, David Jensen, and Rachel Keen—for many encouraging and inspirational discussions. Their comments and feedback significantly contributed to the form of this document. I would especially

In this paper we study the learning of affordances through self-experimentation. We study the learning of local visual descriptors that anticipate the success of a given action executed upon an object. Consider, for instance, the case of grasping. Although graspable is a property of the whole object, the grasp action will only succeed if applied in the right part of the object. We propose an algorithm to learn local visual descriptors of good grasping points based on a set of trials performed by the robot. The method estimates the probability of a successful action (grasp) based on simple local features. Experimental results on a humanoid robot illustrate how our method is able to learn descriptors of good grasping points and to generalize to novel objects based on prior experience.

Abstract. This paper introduces a behavior-grounded approach to representing and learning the affordances of tools by a robot. The affordance representation is learned during a behavioral babbling stage in which the robot randomly chooses different exploratory behaviors, applies them to the tool, and observes their effects on environmental objects. As a result of this exploratory procedure, the tool representation is grounded in the behavioral and perceptual repertoire of the robot. Furthermore, the representation is autonomously testable and verifiable by the robot as it is expressed in concrete terms (i.e., behaviors) that are directly available to the robot’s controller. The tool representation described here can also be used to solve tool-using tasks by dynamically sequencing the exploratory behaviors which were used to explore the tool based on their expected outcomes. The quality of the learned representation was tested on extension-of-reach tasks with rigid tools. 1

Humans use a set of exploratory procedures to examine object properties through grasping and touch. Our goal is to exploit similar methods with a humanoid robot to enable developmental learning about manipulation. We use a compliant robot hand to find objects without prior knowledge of their presence or location, and then tap those objects with a finger. This behavior lets the robot generate and collect samples of the contact sound produced by impact with that object. We demonstrate the feasibility of recognizing objects by their sound, and relate this to human performance under situations analogous to that of the robot. 1.

Abstract — The efficient acquisition and generalization of skills for manual tasks requires that a robot be able to perceive and control the important aspects of an object while ignoring irrelevant factors. For many tasks involving everyday toollike objects, detection and control of the distal end of the object is sufficient for its use. For example, a robot could pour a substance from a bottle by controlling the position and orientation of the mouth. Likewise, the canonical tasks associated with a screwdriver, hammer, or pen rely on control of the tool’s tip. In this paper, we present methods that allow a robot to autonomously detect and control the tip of a tool-like object. We also show results for modeling the appearance of this important type of task relevant feature. 1 I.