A major challenge for the realization of intelligent robots is to supply them with cognitive abilities in order to allow ordinary users to program them easily and intuitively. One way of such programming is teaching work tasks by interactive demonstration. To make this effective and convenient for the user, the machine must be capable to establish a common focus of attention and be able to use and integrate spoken instructions, visual perceptions, and non-verbal clues like gestural commands. We report progress in building a hybrid architecture that combines statistical methods, neural networks, and finite state machines into an integrated system for instructing grasping tasks by man-machine interaction. The system combines the GRAVIS-robot for visual attention and gestural instruction with an intelligent interface for speech recognition and linguistic interpretation, and an modality fusion module to allow multi-modal task-oriented man-machine communication with respect to dextrous robot manipulation of objects. 1

Abstract not found

Abstract — One of the challenges for the realization of future intelligent robots is to design architectures which make user instruction of work tasks by interactive demonstration effective and convenient. A key prerequisite for enhancement of robot learning beyond the level of low-level skill acquisition is situated multi-modal communication. Currently, most existing robot platforms still have to advance to make the development of an integrated learning architecture feasible. We report on the status of the Bielefeld GRAVIS-robot architecture that combines statistical methods, neural networks, and finite state machines into an integrated system for instructing grasping tasks by human-machine interaction. It combines visual attention and gestural instruction with an intelligent interface for speech recognition and linguistic interpretation and a modality fusion module to allow multi-modal task-oriented communication. It further integrates imitation of human hand postures to allow flexible grasping of every-day objects. With respect to this platform, we sketch the concept of a learning architecture based on several interlocking levels with the goal to demonstrate speech-supported imitation learning of grasping. I.

CONFIDENTIAL. Limited circulation. For review only Using touch to localize flexible materials during manipulation

filtering to interpret tactile data

Creating force domain behavior requires control processes that optimize manipulator contact configuration based on the forces that can be applied. In the case of grasping, contact configuration must be optimized for grasp quality measures. If robust controllers can be defined that converge to good grasp configurations, even over limited domains of attraction, then these controllers can be sequenced or combined to create robust behavior over larger domains. This chapter focuses on defining controllers that control grasp quality. Starting with Coelho’s force and moment residual control primitives, a composite grasp controller is defined that executes both of these control primitives concurrently [60]. Next, a hybrid position and force controller is defined that acts in concert with the grasp controller to slide the contacts to good grasp configurations [61]. Finally, the set of potential grasp controllers is expanded by allowing controllers to be parameterized by virtual contacts that correspond to contact groups [57]. Experimental results are presented that demonstrate these controllers to be a practical and effective way of synthesizing grasps in poorly modeled domains. 4.1 Background The control-based approach to solving force-domain problems taken in this thesis rests heavily upon Coelho’s force and moment residual controllers. These controllers minimize grasp error functions by displacing contacts on the surface of an object in response to local tactile feedback at the contacts. This section describes different approaches to tactile sensing and a method for displacing contacts on the surface of an object that acquires tactile feedback, called probing. Next, Coelho’s force residual and moment residual controllers that displace contacts into quality grasp configurations based on local tactile feedback are described. 4.1.1 Sensing for Grasp Control Grasp controllers assume that it is possible to sense local surface geometry in the neighborhood of each contact. This may be accomplished in a number of ways. For example, it may be possible to use computer vision to extract the relationship between object surface and grasp contact [1]. Notice that this vision problem is significantly easier than the general problem of reconstructing full object geometry. Unfortunately, it can be difficult to use vision for the purpose of “tactile sensing ” without placing

Abstract—Dynamic tactile sensing is a fundamental ability for recognizing materials and objects. However, while humans are born with partially developed dynamic tactile sensing and master this skill quickly, today’s robots remain in their infancy. The development of such a sense requires not only better sensors, but also the right algorithms to deal with these sensors ’ data. For example, when classifying a material based on touch, the data is noisy, high-dimensional and contains irrelevant signals as well as essential ones. Few classification methods from machine learning can deal with such problems. In this paper, we propose an efficient approach to inferring suitable lower-dimensional representations of the tactile data. In order to classify materials based on only the sense of touch, these representations are autonomously discovered using visual information of the surfaces during training. However, accurately pairing vision and tactile samples in real robot applications is a difficult problem. The proposed approach therefore works with weak pairings between the modalities. Experiments show that the resulting approach is very robust and yields significantly higher classification performance based on only dynamic tactile sensing. I.