This paper provides a tutorial introduction to visual servo control of robotic manipulators. Since the topic spans many disciplines our goal is limited to providing a basic conceptual framework. We begin by reviewing the prerequisite topics from robotics and computer vision, including a brief review of coordinate transformations, velocity representation, and a description of the geometric aspects of the image formation process. We then present a taxonomy of visual servo control systems. The two major classes of systems, position-based and image-based systems, are then discussed. Since any visual servo system must be capable of tracking image features in a sequence of images, we include an overview of feature-based and correlation-based methods for tracking. We conclude the tutorial with a number of observations on the current directions of the research field of visual servo control. 1 Introduction Today there are over 800,000 robots in the world, mostly working in factory environment...

. Visual servoing, using image-based control or positionbased control, generally gives satisfactory results. However, in some cases, convergence and stability problems may occur. The aim of this paper is to emphasize these problems by considering an eye-in-hand system and a positioning task with respect to a static target which constrains the six camera degrees of freedom. To appear in: The Confluence of Vision and Control, Lecture Notes in Control and Informations Systems, Springer-Verlag, 1998. 1 Introduction The two classical approaches of visual servoing (that is image-based control and position-based control) are different in the nature of the inputs used in their respective control schemes [28,10,14]. Even if the resulting robot behaviors thus also differ, both approaches generally give satisfactory results: the convergence to the desired position is reached, and, thanks to the closed-loop used in the control scheme, the system is stable, and robust with respect to camera calib...

A major difficulty for 3D human body tracking from monocular image sequences is the near non-observability of kinematic degrees of freedom that generate motion in depth. For known link (body segment) lengths, the strict non-observabilities reduce to twofold ‘forwards/backwards flipping ’ ambiguities for each link. These imply 2 # links formal inverse kinematics solutions for the full model, and hence linked groups of O(2 # links) local minima in the model-image matching cost function. Choosing the wrong minimum leads to rapid mistracking, so for reliable tracking, rapid methods of investigating alternative minima within a group are needed. Previous approaches to this have used generic search methods that do not exploit the specific problem structure. Here, we complement these by using simple kinematic reasoning to enumerate the tree of possible forwards/backwards flips, thus greatly speeding the search within each linked group of minima. Our methods can be used either deterministically, or within stochastic ‘jump-diffusion ’ style search processes. We give experimental results on some challenging monocular human tracking sequences, showing how the new kinematic-flipping based sampling method improves and complements existing ones.

In this paper, we propose a new approach to vision-based robot control, called 2 1/2 D visual servoing, which avoids the respective drawbacks of classical positionbased and image-based visual servoing. Contrary to the position-based visual servoing, our scheme does not need any geometric 3D model of the object. Furthermore and contrary to image-based visual servoing, our approach ensures the convergence of the control law in the whole task space. 2 1/2 D visual servoing is based on the estimation of the partial camera displacement from the current to the desired camera poses at each iteration of the control law. Visual features and data extracted from the partial displacement allow us to design a decoupled control law controlling the six camera d.o.f. The robustness of our visual servoing scheme with respect to camera calibration errors is also analyzed: the necessary and sufficient conditions for local asymptotic stability are easily obtained. Then, due to the simple structure of the ...

. We describe a novel method of obtaining a fixation point on a moving object for a real-time gaze control system. The method makes use of a real-time implementation of a corner detector and tracker and reconstructs the image position of the desired fixation point from a cluster of corners detected on the object using the affine structure available from two or three views. The method is fast, reliable, viewpoint invariant, and insensitive to occlusion and/or individual corner dropout or reappearance. We compare two- and three-dimensional forms of the algorithm, present results for the method in use with a high performance head/eye platform, and compare the results with two naive fixation methods. Keywords: active vision, head/eye platform, gaze control, tracking, fixation, affine structure 1. Introduction One of the tenets of the active vision paradigm is that fixation over time -- the process of centering and holding gaze on a point in the environment -- can make many problems in co...

Abstract—This paper addresses the problem of coordinating multiple spacecraft to fly in tightly controlled formations. The main contribution of the paper is to introduce a coordination architecture that subsumes leader-following, behavioral, and virtual-structure approaches to the multiagent coordination problem. The architecture is illustrated through a detailed application of the ideas to the problem of synthesizing a multiple spacecraft interferometer in deep space. Index Terms—Control architecture, coordinated control, interferometry, spacecraft formation flying. I.

In this paper, several vision-based robot control methods are classified following an analogy with well known minimization methods. Comparing the rate of convergence between minimization algorithms helps us to understand the difference of performance of the control schemes. In particular, it is shown that standard vision-based control methods have in general low rates of convergence. Thus, the performance of vision-based control could be improved using schemes which perform like the Newton minimization algorithm that has a high convergence rate. Unfortunately, the Newton minimization method needs the computation of second derivatives that can be ill-conditioned causing convergence problems. In order to solve these problems, this paper proposes two new control schemes based on efficient second-order minimization techniques.

This paper attempts to present a comprehensive summary of research results in the use of visual information to control robot manipulators and related mechanisms. An extensive bibliography is provided which also includes important papers from the elemental disciplines upon which visual servoing is based. The research results are discussed in terms of historical context, commonality of function, algorithmic approach and method of implementation. 1 Introduction This paper presents the history, and reviews current research into the use of visual information for the control of robot manipulators and mechanisms. Visual control of manipulators promises substantial advantages when working with targets whose position is unknown, or with manipulators which may be flexible or inaccurate. The reported use of visual information to guide robots, or more generally mechanisms, is quite extensive and encompasses manufacturing applications, teleoperation, missile tracking cameras, fruit picking as well...

In this paper, we propose a new approach to visionbased robot control, called 2-1/2-D visual servoing, which avoids the respective drawbacks of classical position-based and imagebased visual servoing. Contrary to the position-based visual servoing, our scheme does not need any geometric three-dimensional (3-D) model of the object. Furthermore and contrary to imagebased visual servoing, our approach ensures the convergence of the control law in the whole task space. 2-1/2-D visual servoing is based on the estimation of the partial camera displacement from the current to the desired camera poses at each iteration of the control law. Visual features and data extracted from the partial displacement allow us to design a decoupled control law controlling the six camera d.o.f. The robustness of our visual servoing scheme with respect to camera calibration errors is also analyzed: the necessary and sufficient conditions for local asymptotic stability are easily obtained. Then, due to the simple structure of the system, sufficient conditions for global asymptotic stability are established. Finally, experimental results with an eye-in-hand robotic system confirm the improvement in the stability and convergence domain of the 2-1/2-D visual servoing with respect to classical position-based and image-based visual servoing.

Visual servoing, i.e. the use of the vision sensor in feedback control, has been of increasing interest. A fair amount of work has been done in applications in autonomous driving, manipulation, mobile robot navigation and surveillance. However, the theoretical and analytical aspects of the problem have not received much attention. Furthermore, the problem of estimation from the vision measurements has been considered separately from the design of the control strategies. Instead of addressing the pose estimation and control problems separately, we attempt to characterize the types of control tasks which can be achieved using only quantities directly measurable in the image, bypassing the pose estimation phase. We consider the navigation task for a nonholonomic ground mobile base tracking an arbitrarily shaped continuous ground curve. This tracking problem is formulated as one of controlling the shape of the curve in the image plane. We study the controllability of the system characteriz...