This paper provides a tutorial introduction to visual servo control of robotic manipulators. Since the topic spans many disciplines our goal is limited to providing a basic conceptual framework. We begin by reviewing the prerequisite topics from robotics and computer vision, including a brief review of coordinate transformations, velocity representation, and a description of the geometric aspects of the image formation process. We then present a taxonomy of visual servo control systems. The two major classes of systems, position-based and image-based systems, are then discussed. Since any visual servo system must be capable of tracking image features in a sequence of images, we include an overview of feature-based and correlation-based methods for tracking. We conclude the tutorial with a number of observations on the current directions of the research field of visual servo control. 1 Introduction Today there are over 800,000 robots in the world, mostly working in factory environment...

this paper, we applied the proposed methodology to various problems such as occlusions avoidance, constraints on the field of view (i.e., keeping an object inside view), 3D contact in a cluttered environment (i.e., obstacle avoidance). This method as been previously used for singularities and joint limits avoidance [11]. A similar approach has been proposed by Nelson and Khosla. It consists of minimizing an objective function which realizes a compromise between the visual task (a target tracking using a camera mounted on the end effector of a manipulator) and the avoidance of kinematic singularities, joint limits singularities but also with some other constraints on the field of view, the focus measure [12]. This function is used by exploiting the robot degrees of freedom which are redundant with respect to the visual task. However, the resulting camera motions can produce major perturbations in the visual servoing since they are generally not compatible with the regulation to zero of the selected image features. The next section of this paper, taken from [11], recalls the application of the task function approach to visual servoing and the expression of the resulting control law. Section 3 describes the approach proposed to dynamic sensor planning. We finally present real time experimental results dealing with various robotic tasks. These results have been obtained using an eye-in-hand system composed of a camera mounted on the end-effector of a six d.o.f Zebra Zero robot.

The Chimera Methodology is a new software engineering paradigm which addresses the problem of developing dynamically reconfigurable and reusable real-time software. The foundation of the Chimera methodology is the port-based object model of a reusable software component. The model is obtained by applying the port-automaton formal computational model to object-based design. Global state variable table real-time communication is used to integrate port-based objects, which eliminates the need for writing and debugging glue code. The Chimera real-time operating system provides tools to support the software models defined by the Chimera methodology, so that real-time software can be executed predictably using common real-time scheduling algorithms. A hypermedia user interface has been designed to allow users to easily assemble the real-time software components that are designed based on the Chimera methodology. Use of the methodology can result in a significant decrease the development time and cost of real-time applications.

We present a working implementation of a dynamics based architecture for visual sensing. This architecture provides field rate estimates of the positions and velocities of two independent falling balls in the face of repeated visual occlusions and departures from field of view. The practical success of this system can be attributed to the feedback interconnection between two strongly nonlinear dynamical systems: a novel "triangulating" state estimator; and an image plane window controller. We detail the architecture of this active sensor, provide data documenting its performance, and offer an initial analysis of its soundness in the form of a convergence proof for (a simpler version of) the estimator and a boundedness proof for (a somewhat idealized version of) the manager. 1 Introduction We have built a three degree of freedom robot that bats two balls into simultaneous stable periodic vertical trajectories that commonly persist for the better part of an hour [20]. The juggling algor...

Vision feedback control loop techniques are efficient for a great class of applications but they come up against difficulties when the initial and desired positions of the camera are distant. In this paper we propose a new approach to resolve these difficulties by planning trajectories in the image. Constraints such that the object remains in the camera field of view can thus be taken into account. Furthermore, using this process, current measurement always remain close to their desired value and a control by Imagebased Servoing ensures the robustness with respect to modeling errors. We apply our method when object dimension are known or not and/or when the calibration parameters of the camera are well or badly estimated. Finally, real time experimental results using a camera mounted on the end effector of a six d-o-f robot are presented.

Force and vision sensors provide complementary information, yet they are fundamentally different sensing modalities. This implies that traditional sensor integration techniques that require common data representations are not appropriate for combining the feedback from these two disparate sensor. In this paper, we introduce the concept of vision and force sensor resolvability as a means of comparing the ability of the two sensing modes to provide useful information during robotic manipulation tasks. By monitoring the resolvability of the two sensing modes with respect to the task, the information provided by the disparate sensors can be seamlessly assimilated during task execution. A nonlinear force/vision servoing algorithm that uses force and vision resolvability to switch between sensing modes is proposed. The advantages of the assimilation technique is demonstrated during contact transitions between a stiff manipulator and rigid environment, a system configuration that easily becom...

This paper presents an original solution to the camera control problem in a virtual environment. Our objective is to present a general framework that allows the automatic control of a camera in a dynamic environment. The proposed method is based on the image-based control or visual servoing approach. It consists in positioning a camera according to the information perceived in the image. This is thus a very intuitive approach of animation. To be able to react automatically to modifications of the environment, we also considered the introduction of constraints into the control. This approach is thus adapted to highly reactive contexts (virtual reality, video games). Numerous examples dealing with classic problems in animation are considered within this framework and presented in this paper. Key words: Automatic camera motion, Automatic cinematography, Visual servoing, Animation 1Overview Issues. There are numerous issues related to the control of a camera in a virtual environment. Ty...

This paper presents a new application for image-based visual servoing: computer graphics animation. Indeed, the control of a virtual camera in virtual environment is not a trivial problem and usually required skilled operators. Visual servoing, a now well known technique in robotics and computer vision, consists in positioning a camera according to the informations perceived in the images. Using this method within computer graphics context leads to a very intuitive approach of animation. Furthermore, in that case a full knowledge about the scene is available. It allows to easily introduce constraints within the control law in order to react automatically to modifications of the environment. In this paper, we apply this approach in two different contexts: highly reactive applications (virtual reality, video games) and the control of humanoid avatars.

We consider the problem of planning sensor strategies that enable a sensor to be automatically con g-ured for robot tasks. In this paper we present robust and e cient algorithms for computing the regions from which a sensor has unobstructed or partially obstructed views of a target in a goal. We apply these algorithms to the Error Detection and Recovery problem of recognizing whether a goal or failure region has been achieved. Based on these methods and strategies for visually-cued camera control, we have built a robot surveillance system in which one mobile robot navigates to a viewing position from which it has an unobstructed view of a goal region, and then uses visual recognition to detect when a speci c target has entered the room. 1

We present an O(n log for computing the optimal robot motion that maintains lineof -sight visibility between a target moving inside a polygon with n vertices which may contain holes. The motion is optimal for the tracking robot (the observer) in the sense that the target either remains visible for the longest possible time, or it is captured by the observer in the minimum time when feasible. Thus, the algorithm maximizes the minimum time-to-escape. Our algorithm assumes that the target moves along a known path. Thus, it is an off-line algorithm. Our theoretical results for the algorithm's runtime assume that the target is moving along a shortest path from its source to its destination. This assumption, however is not required to prove the optimality of the computed solution, hence the algorithm remains correct for the general case.