To my mother and father, and to my wife. iii Acknowledgements I would like to take this opportunity to express my gratitude to the people who have helped and supported me during my Ph.D. program. First and foremost, I am particularly grateful to my adviser, Professor Demetri Terzopoulos. It was his guidance, encouragement and collaboration that lead me along the bumpy road of Ph.D. study to this final accomplishment. I am so fortu-nate to have had the experience of research and study with him for the past five years, which has changed me and will be influencing me for the rest of my life. Next, I would like to thank Professors Ken Perlin, Davi Geiger, Yann LeCun, Denis Zorin and Chris Bregler for serving on my proposal and dissertation com-mittees. Special thanks go to Ken for his insightful opinions and suggestions on my research work. I owe a lot to my colleagues and lab mates, among them Mauricio Plaza who worked on the reconstructed Penn Station model with me, Alex Vasilescu, Sung-Hee Lee and Evgueni Parilov who shared their ideas, opinions, discussion and jokes with me, and everybody at the Media Research Lab for the discussions, laughter, food and drink. The research reported herein was supported in part by grants from the Defense iv

Abstract — Estimating the traversability of terrain in an unstructured outdoor environment is a core functionality for autonomous robot navigation. While general-purpose sensing can be used to identify the existence of terrain features such as vegetation and sloping ground, the traversability of these regions is a complex function of the terrain characteristics and vehicle capabilities, which makes it extremely difficult to characterize a priori. Moreover, it is difficult to find general rules which work for a wide variety of terrain types such as trees, rocks, tall grass, logs, and bushes. As a result, methods which provide traversability estimates based on predefined terrain properties such as height or shape will be unlikely to work reliably in unknown outdoor environments. Our approach is based on the observation that traversability in the most general sense is an affordance which is jointly determined by the vehicle and its environment. We describe a novel on-line learning method which can make accurate predictions of the traversability properties of complex terrain. Our method is based on autonomous training data collection which exploits the robot’s experience in navigating its environment to train classifiers without human intervention. This is in contrast to other learning methods in which training data is collected manually. We have implemented and tested our traversability learning method on an ummaned ground vehicle (UGV) and evaluated its performance in several realistic outdoor environments. The experiments quantify the benefit of our on-line traversability learning approach. I.

Abstract — Recent work in the object recognition community has yielded a class of interest point-based features that are stable under significant changes in scale, viewpoint, and illumination, making them ideally suited to landmark-based navigation. Although many such features may be visible in a given view of the robot’s environment, only a few such features are necessary to estimate the robot’s position and orientation. In this paper, we address the problem of automatically selecting, from the entire set of features visible in the robot’s environment, the minimum (optimal) set by which the robot can navigate its environment. Specifically, we decompose the world into a small number of maximally sized regions such that at each position in a given region, the same small set of features is visible. We introduce a novel graph theoretic formulation of the problem and prove that it is NP-complete. Next, we introduce a number of approximation algorithms and evaluate them on both synthetic and real data. I.

Man made indoors environments posses regularities which can be efficiently exploited in automated model acquisition by means of visual sensing. In this context we propose an approach for inferring a topological model of an environment from images or the video stream captured by a mobile robot during exploration. The proposed model consists of a set of locations and neighbourhood relationships between them. Initially each location in the model is represented by a collection of similar, temporally adjacent views, with the similarity defined according to a simple appearance based distance measure. The sparser representation is obtained in a subsequent learning stage by means of Learning Vector Quantization (LVQ). The quality of the model is tested in the context of qualitative localization scheme by

Abstract—We present a distributed vision-based architecture for smart robotics that is composed of multiple control loops, each with a specialized level of competence. Our architecture is subsumptive and hierarchical, in the sense that each control loop can add to the competence level of the loops below, and in the sense that the loops can present a coarse-to-fine gradation with respect to vision sensing. At the coarsest level, the processing of sensory information enables a robot to become aware of the approximate location of an object in its field of view. On the other hand, at the finest end, the processing of stereo information enables a robot to determine more precisely the position and orientation of an object in the coordinate frame of the robot. The processing in each module of the control loops is completely independent and it can be performed at its own rate. A control Arbitrator ranks the results of each loop according to certain confidence indices, which are derived solely from the sensory information. This architecture has clear advantages regarding overall performance of the system, which is not affected by the “slowest link, ” and regarding fault tolerance, since faults in one module does not affect the other modules. At this time we are able to demonstrate the utility of the architecture for stereoscopic visual servoing. The architecture has also been applied to mobile robot navigation and can easily be extended to tasks such as “assembly-on-the-fly.” Index Terms—Assembly-on-the-fly, automation, computer vision, distributed architectures, robotics, vision-based architecture,

Abstract—This paper presents a novel coarse-to-fine global localization approach inspired by object recognition and text retrieval techniques. Harris–Laplace interest points characterized by scale-invariant transformation feature descriptors are used as natural landmarks. They are indexed into two databases: a location vector space model (LVSM) and a location database. The localization process consists of two stages: coarse localization and fine localization. Coarse localization from the LVSM is fast, but not accurate enough, whereas localization from the location database using a voting algorithm is relatively slow, but more accurate. The integration of coarse and fine stages makes fast and reliable localization possible. If necessary, the localization result can be verified by epipolar geometry between the representative view in the database and the view to be localized. In addition, the localization system recovers the position of the camera by essential matrix decomposition. The localization system has been tested in indoor and outdoor environments. The results show that our approach is efficient and reliable. Index Terms—Coarse-to-fine localization, scale-invariant features, vector space model, visual vocabulary.

This paper presents a novel method for mobile robot localization using visual appearance features. A multidimensional-histogram is used to describe the global appearance features of an image such as colors, edge density, gradient magnitude, textures and so on. The matching of histograms determines the location of the robot. The method has been evaluated in an indoor environment, and the system correctly determines the localization of 82.9 % of the input scene images. 1.

Abstract—Autonomous robot navigation in unstructured outdoor environments is a challenging area of active research. The navigation task requires identifying safe, traversable paths which allow the robot to progress toward a goal while avoiding obstacles. One approach is to apply Machine Learning techniques that accomplish near to far learning by augmenting near-field Stereo to identify safe terrain and obstacles in the far field. Some mechanism for applying past learned experience to the active navigation task is crucial for effective far-field classification. We introduce a new method for long-term learning in the robot navigation task by selecting a subset of previously learned linear binary classifiers. We then combine their output to produce a final classification for a new image. Techniques for efficient selection of models, as well as the combination of their output, are addressed. We evaluate the performance of our technique on three fully labeled datasets, and show that our technique outperforms several baseline techniques that do not leverage past experience. I.

This work surveys the major methods for model-based active sensing in robotics. Active sensing in robotics incorporates the following aspects: (i) where to position sensors, and (ii) how to make decisions for next actions, in order to maximize information gain and minimize costs.

We present a simple approach for vision-based path following for a mobile robot. Based upon a novel concept called the funnel lane, the coordinates of feature points during the replay phase are compared with those obtained during the teaching phase in order to determine the turning direction. Increased robustness is achieved by coupling the feature coordinates with odometry information. The system requires a single off-the-shelf, forward-looking camera with no calibration (either external or internal, including lens distortion). Implicit calibration of the system is needed only in the form of a single controller gain. The algorithm is qualitative in nature, requiring no map of the environment, no image Jacobian, no homography, no fundamental matrix, and no assumption about a flat ground plane. Experimental results demonstrate the capability of real-time autonomous navigation in both indoor and outdoor environments, on flat, slanted, and rough terrain with dynamic occluding objects for distances of hundreds of meters. We also demonstrate that the same approach works with wide-angle and omnidirectional cameras with only slight modification.