This paper provides a tutorial introduction to visual servo control of robotic manipulators. Since the topic spans many disciplines our goal is limited to providing a basic conceptual framework. We begin by reviewing the prerequisite topics from robotics and computer vision, including a brief review of coordinate transformations, velocity representation, and a description of the geometric aspects of the image formation process. We then present a taxonomy of visual servo control systems. The two major classes of systems, position-based and image-based systems, are then discussed. Since any visual servo system must be capable of tracking image features in a sequence of images, we include an overview of feature-based and correlation-based methods for tracking. We conclude the tutorial with a number of observations on the current directions of the research field of visual servo control. 1 Introduction Today there are over 800,000 robots in the world, mostly working in factory environment...

In contrast to the traditional approach, visual recognition is formulated as one of matching appearance rather than shape. For any given robot vision task, all possible appearance variations define its visual workspace. A set of images is obtained by coarsely sampling the workspace. The image set is compressed to obtain a low-dimensional subspace, called the eigenspace, in which the visual workspace is represented as a continuous appearance manifold. Given an unknown input image, the recognition system first projects the image to eigenspace. The parameters of the vision task are recognized based on the exact location of the projection on the appearance manifold. Efficient algorithms for finding the closest manifold point are discussed. The proposed appearance representation has several applications in robot vision. As examples, a precise visual positioning system, a real-time visual tracking system, and a real-time temporal inspection system are described. The performance of these syst...

Much of the previous work on hand-eye coordination has emphasized the reconstructive aspects of vision. Recently, techniques that avoid explicit reconstruction by placing visual feedback into a control loop have been developed. When properly defined, these methods lead to calibration insensitive hand-eye coordination. In this article, recent work on projective geometry as applied to vision is used to extend this paradigm in two ways. First, it is shown how results from projective geometry can be used to perform online calibration. Second, results on projective invariance are used to define setpoints for visual control that are independent of viewing location. These ideas are illustrated through a number of examples and have been tested on an implemented system. 1 Introduction The goal of hand-eye coordination research is to provide a general-purpose mechanism for controlling a robotic mechanism from visual inputs. A natural paradigm to follow in this quest is to emphasize the reconst...

This paper attempts to present a comprehensive summary of research results in the use of visual information to control robot manipulators and related mechanisms. An extensive bibliography is provided which also includes important papers from the elemental disciplines upon which visual servoing is based. The research results are discussed in terms of historical context, commonality of function, algorithmic approach and method of implementation. 1 Introduction This paper presents the history, and reviews current research into the use of visual information for the control of robot manipulators and mechanisms. Visual control of manipulators promises substantial advantages when working with targets whose position is unknown, or with manipulators which may be flexible or inaccurate. The reported use of visual information to guide robots, or more generally mechanisms, is quite extensive and encompasses manufacturing applications, teleoperation, missile tracking cameras, fruit picking as well...

This article describes the theory and implementation of a system that positions a robot manipulator using visual information from two cameras. The system simultaneously tracks the robot end-effector and visual features used to define goal positions. An error signal based on the visual distance between the end-effector and the target is defined and a control law that moves the robot to drive this error to zero is derived. The control law has been integrated into a system that performs tracking and stereo control on a single processor with no special purpose hardware at real-time rates. Experiments with the system have shown that the controller is so robust to calibration error that the cameras can be moved several centimeters and rotated several degrees while the system is running with no adverse effects. Introduction Over the past several years, the use of sensor feedback in robotic systems has been an active area of research. However, the development of visual feedback mechanisms has...

We propose a method for visual control of a robotic system which does not require the formulation of an explicit calibration between image space and the world coordinate system. Calibration is known to be a difficult and error prone process. By extracting control information directly from the image, we free our technique from the errors normally associated with a fixed calibration. We demonstrate this by performing a peg-in-hole alignment using an uncalibrated camera to control the positioning of the peg. The algorithm utilizes feedback from a simple geometric effect, rotational invariance, to control the positioning servo loop. The method uses an approximation to the Image Jacobian to provide smooth, near-continuous control. 1 INTRODUCTION In many real world applications, there exists a need to align one object with another. Many researchers have concentrated their efforts on recovering the true position (the world coordinates) of the object to be manipulated. They used these results...

Algorithms for full 3D robotic visual tracking of moving targets whose motion is 3D and consists of translational and rotational components are presented. The objective of the system is to track selected features on moving objects and to place their projections on the image plane at desired positions by appropriate camera motion. The most important characteristics of the proposed algorithms are the use of a single camera mounted on the end-effector of a robotic manipulator (eye-in-hand configuration), and the fact that these algorithms do not require accurate knowledge of the relative distance of the target object from the camera frame. The detection of motion is based on a cross-correlation technique known as Sum-of-Squares Differences (SSD) algorithm. The camera model used introduces a number of parameters that are estimated on-line, further reducing the algorithms' reliance on precise calibration of the system. An adaptive control algorithm compensates for modeling errors, tracking ...

There are many design factors and choices when mounting a vision system for robot control. Such factors may include the kinematic and dynamic characteristics in the robot's degrees of freedom (DOF), which determine at what velocities and fields-of-view a camera can achieve. Another factor is that additional motion components (such as pan-tilt units) are often mounted on a robot and introduce synchronization problems. When a task does not require visually servoing every robot DOF, the designer must choose which ones to servo. Questions then arise as to what roles, if any, do the remaining DOF play in the task. Without an analytical framework, the designer resorts to intuition and try-and-see implementations. This paper presents a frequency-based framework that identifies the parameters that factor into tracking. This framework gives design insight which was then used to synthesize a control law that exploits the kinematic and dynamic attributes of each DOF. The resulting multi-input multi-output control law, which we call partitioning, defines an underlying joint-coupling to servo camera motions. The net effect is that by employing both visual and kinematic feedback loops, a robot can quickly position and orient a camera in a large assembly workcell. Real-time experiments tracking people and robot hands are presented using a 5-DOF hybrid (3-DOF Cartesian gantry plus 2-DOF pan-tilt unit) robot.

We propose two methods for visual control of a robotic system which do not require the formulation of an explicit calibration between image space and the world coordinate system. Calibration is known to be a difficult and error prone process. By extracting control information directly from the image, we free our techniques from the errors normally associated with a fixed calibration. The two algorithms we propose both utilize feedback from a simple geometric effect, rotational invariance, to control the positioning servo loop. We attach a camera system to a robot such that the camera system and the robot's gripper rotate simultaneously. We also constrain the camera to lie in a position where it can observe the gripper's rotational axis. As the camera system rotates about the gripper's rotational axis, the circular path traced out by a point-like feature projects to an elliptical path in image space. We gather the projected feature points over part of a rotation (=2 radians) ...

Image-based servoing systems are often used to track moving targets and their underlying control architecture is a regulation of the image. This regulation is a function of rigid camera-to-target geometric constraints. Satisfying such constraints requires that the robot motors have sufficient velocity bandwidths, and often these bandwidths are limited. This paper lays down the foundation for a partitioned controller. Such a controller would coordinate a camera's DOF into a synergistic move to overcome bandwidth limitations. Tracking experiments are shown on a custom designed 5 DOF gantry robot which highlight the limitations of regulator-based control, as well as show how partitioning can be used to achieve more robust control.