This paper provides a tutorial introduction to visual servo control of robotic manipulators. Since the topic spans many disciplines our goal is limited to providing a basic conceptual framework. We begin by reviewing the prerequisite topics from robotics and computer vision, including a brief review of coordinate transformations, velocity representation, and a description of the geometric aspects of the image formation process. We then present a taxonomy of visual servo control systems. The two major classes of systems, position-based and image-based systems, are then discussed. Since any visual servo system must be capable of tracking image features in a sequence of images, we include an overview of feature-based and correlation-based methods for tracking. We conclude the tutorial with a number of observations on the current directions of the research field of visual servo control. 1 Introduction Today there are over 800,000 robots in the world, mostly working in factory environment...

This paper attempts to present a comprehensive summary of research results in the use of visual information to control robot manipulators and related mechanisms. An extensive bibliography is provided which also includes important papers from the elemental disciplines upon which visual servoing is based. The research results are discussed in terms of historical context, commonality of function, algorithmic approach and method of implementation. 1 Introduction This paper presents the history, and reviews current research into the use of visual information for the control of robot manipulators and mechanisms. Visual control of manipulators promises substantial advantages when working with targets whose position is unknown, or with manipulators which may be flexible or inaccurate. The reported use of visual information to guide robots, or more generally mechanisms, is quite extensive and encompasses manufacturing applications, teleoperation, missile tracking cameras, fruit picking as well...

Whenever a sensor is mounted on a robot hand it is important toknow the relationship between the sensor and the hand. The problem of determining this relationship is referred to as the hand-eye calibration problem. Hand-eye calibration is important in at least two types of tasks: (i) map sensor centered measurements into the robot workspace frame and (ii) allow the robot to precisely move the sensor. In the past some solutions were proposed in the particular case of the sensor being a TV camera. With almost no exception, all existing solutions attempt to solve a homogeneous matrix equation of the form AX = XB. This paper has the following main contributions. First we show that there are two possible formulations of the hand-eye calibration problem. One formulation is the classical one that we just mentioned. A second formulation takes the form of the following homogeneous matrix equation: MY = M 0 YB. The advantage of the latter formulation is that the extrinsic and intrinsic parameters of the camera need not be made explicit. Indeed, this formulation directly uses the 3 4 perspective matrices (M and M 0) associated with 2 positions of the camera with respect to the calibration frame. Moreover, this formulation together with the classical one cover a wider range of camera-based sensors to be calibrated with respect to the robot hand: single scan-line cameras, stereo heads, range nders, etc. Second, we develop a common mathematical framework to solve for the hand-eye calibration problem using either of the two formulations. We represent rotation by a unit quaternion. We present two methods, (i) a closed-form solution for solving for rotation using unit quaternions and then solving for translation and (ii) a non-linear technique for simultaneously solving for rotation and translation. Third, we perform a stability analysis both for our two methods and for the classical linear method developed by Tsai & Lenz [TL89]. This analysis allows the comparison of the three methods. In the light of this comparison, the non-linear optimization method, that solves for rotation and translation simultaneously, seems to be the most robust one with respect to noise and to measurement errors.

We address in this paper the problem of self-calibration and metric reconstruction (up to a scale) from one unknown motion of an uncalibrated stereo rig, assuming the coordinates of the principal point of each camera are known (This assumption is not necessary if one more motion is available). The epipolar constraint is first formulated for two uncalibrated images. The problem then becomes one of estimating unknowns such that the discrepancy from the epipolar constraint, in terms of distances between points and their corresponding epipolar lines, is minimized. The initialization of the unknowns is based on the work of Maybank, Luong and Faugeras on self-calibration of a single moving camera, which requires to solve a set of so-called Kruppa equations. Redundancy of the information contained in a sequence of stereo images makes this method more robust than using a sequence of monocular images. Real data have been used to test the proposed method, and the results obtained are quite goo...

Abstract. In this paper we describe the theory and practice ofself-calibration ofcameras which are fixed in location and may freely rotate while changing their internal parameters by zooming. The basis ofour approach is to make use ofthe so-called infinite homography constraint which relates the unknown calibration matrices to the computed inter-image homographies. In order for the calibration to be possible some constraints must be placed on the internal parameters ofthe camera. We present various self-calibration methods. First an iterative non-linear method is described which is very versatile in terms ofthe constraints that may be imposed on the camera calibration: each ofthe camera parameters may be assumed to be known, constant throughout the sequence but unknown, or free to vary. Secondly, we describe a fast linear method which works under the minimal assumption of zero camera skew or the more restrictive conditions ofsquare pixels (zero skew and known aspect ratio) or known principal point. We show experimental results on both synthetic and real image sequences (where ground truth data was available) to assess the accuracy and the stability ofthe algorithms and to compare the result ofapplying different constraints on the camera parameters. We also derive an optimal Maximum Likelihood estimator for the calibration and the motion parameters. Prior knowledge about the distribution ofthe estimated parameters (such as the location ofthe principal point) may also be incorporated via Maximum a Posteriori estimation. We then identify some near-ambiguities that arise under rotational motions showing that coupled changes ofcertain parameters are barely observable making them indistinguishable. Finally we study the negative effect ofradial distortion in the self-calibration process and point out some possible solutions to it. 1.

In this work, we present a fundamentally new approach to visual servoing using lines. It is based on a theoretical and geometrical study of the main line representations which allowed us to define a new representation, the so-called binormalized Plücker coordinates. They are particularly well suited to visual servoing. Indeed, they allow the definition of a proper image line alignment notion. The control law which realizes such an alignment has moreover several properties: partial decoupling between rotation and translation, analytical inversion of the motion equations and global asymptotic stability conditions. This control law was validated both in simulation and experimentally in the specific case of an orthogonal trihedron.

To relate measurements made by a sensor mounted on a mechanical link to the robot’s coordinate frame, we must first estimate the transformation between these two frames. Many algorithms have been proposed for this so-called hand-eye calibration, but they do not treat the relative position and orientation in a unified way. In this paper, we introduce the use of dual quaternions, which are the algebraic counterpart of screws. Then we show how a line transformation can be written with the dual-quaternion product. We algebraically prove that if we consider the camera and motor transformations as screws, then only the line coefficients of the screw axes are relevant regarding the hand-eye calibration. The dual-quaternion parameterization facilitates a new simultaneous solution for the hand-eye rotation and translation using the singular value decomposition. Real-world performance is assessed directly in the application of hand-eye information for stereo reconstruction, as well as in the positioning of cameras. Both real and synthetic experiments show the superiority of the approach over two other proposed methods.

Abstract not found

Abstract—Vision-aided inertial navigation systems (V-INSs) can provide precise state estimates for the 3-D motion of a vehicle when no external references (e.g., GPS) are available. This is achieved by combining inertial measurements from an inertial measurement unit (IMU) with visual observations from a camera under the assumption that the rigid transformation between the two sensors is known. Errors in the IMU-camera extrinsic calibration process cause biases that reduce the estimation accuracy and can even lead to divergence of any estimator processing the measurements from both sensors. In this paper, we present an extended Kalman filter for precisely determining the unknown transformation between a camera and an IMU. Contrary to previous approaches, we explicitly account for the time correlation of the IMU measurements and provide a figure of merit (covariance) for the estimated transformation. The proposed method does not require any special hardware (such as spin table or 3-D laser scanner) except a calibration target. Furthermore, we employ the observability rank criterion based on Lie derivatives and prove that the nonlinear system describing the IMU-camera calibration process is observable. Simulation and experimental results are presented that validate the proposed method and quantify its accuracy. Index Terms—Extended Kalman filter, inertial measurement unit (IMU)-camera calibration, Lie derivatives, observability of nonlinear systems, vision-aided inertial navigation. I.

In this paper, we first review research results of camera self-calibration achieved in photogrammetry, robotics and computer vision. Then we propose a method for selfcalibration of robotic hand cameras by means of active motion. Through tracking a set of world points of unknown coordinates during robot motion, the internal parameters of the cameras (including distortions), the mounting parameters as well as the coordinates of the world points are estimated. The approach is fully autonomous, in that no initial guesses of the unknown parameters are to be provided from the outside by humans for the solution of a set of nonlinear equations. Sufficient conditions for a unique solution are derived in terms of controlled motion sequences. Methods to improve accuracy and robustness are proposed by means of best model identification and motion planning. Experimental results in both a simulated and a real environments are reported. Key words: self-calibration, hand-cameras, hand-eye calibration...