Human speech is inherently multi-modal, consisting of both audio and visual components. Recently researchers have shown that the incorporation of information about the position of the lips into acoustic speech recognisers enables robust recognition of noisy speech. In the case of Hidden Markov Modelrecognition, we show that this happens because the visual signal stabilises the alignment of states. It is also shown that unadorned lips, both the inner and outer contours, can be robustly tracked in real time on general-purpose workstations. To accomplish this, efficient algorithms are employed which contain three key components: shape models, motion models, and focused colour feature detectors --- all of which are learnt from examples.

This paper addresses some problems in the interpretation of visually observed shapes in motion, both planar and three-dimensional shapes. Mumford (1996), interpreting the "Pattern Theory" developed over a number of years by Grenander (1976), views images as "pure" patterns that have been distorted by a combination of four kinds of degradations. This view applies naturally to the analysis of static, two-dimensional images. The four degradations are given here, together with comments on how they need to be extended to take account of three-dimensional objects in motion.

Abstract—We propose a new image segmentation technique called strings. A string is a variational deformable model that is learned from a collection of example objects rather than built from a priori analytical or geometrical knowledge. As opposed to existing approaches, an object boundary is represented by a one-dimensional multivariate curve in functional space, a feature function, rather than by a point in vector space. In the learning phase, feature functions are defined by extraction of multiple shape and image features along continuous object boundaries in a given learning set. The feature functions are aligned, then subjected to functional principal components analysis and functional principal regression to summarize the feature space and to model its content, respectively. Also, a Mahalanobis distance model is constructed for evaluation of boundaries in terms of their feature functions, taking into account the natural variations seen in the learning set. In the segmentation phase, an object boundary in a new image is searched for with help of a curve. The curve gives rise to a feature function, a string, that is weighted by the regression model and evaluated by the Mahalanobis model. The curve is deformed in an iterative procedure to produce feature functions with minimal Mahalanobis distance. Strings have been compared with active shape models on 145 vertebra images, showing that strings produce better results when initialized close to the target boundary, and comparable results otherwise. Index Terms—Machine learning, deformable models, energy minimization, multivariate statistics, shape analysis, functional data analysis, chemometrics, active shape models. 1

We describe how to teach deformable models to maximize image segmentation correctness based on user-specified criteria, and we present a method for evaluating which criteria work best. We present sectored snakes, a formulation that demonstrably improves upon regular snakes.

The correspondence problem in computer vision is basically a matching task between two or more sets of features. In this paper, we introduce a vectorized image representation, which is a feature-based representation where correspondence has been established with respect to a reference image. The representation consists of two image measurements made at the feature points: shape and texture. Feature geometry, or shape, is represented using the (x; y) locations of features relative to the some standard reference shape. Image grey levels, or texture, are represented by mapping image grey levels onto the standard reference shape. Computing this representation is essentially a correspondence task, and in this paper we explore an automatic technique for "vectorizing" face images. Our face vectorizer alternates back and forth between computation steps for shape and texture, and a key idea is to structure the two computations so that each one uses the output of the other. In addition to descri...

Research at the Computer Vision Laboratory at the University of Maryland has focussed on developing algorithms and systems that can look at humans and recognize their activities in near real-time. Our earlier implementation (the W 4 system) while quite successful, was restricted to applications with a fixed camera. In this paper we present some recent work that removes this restriction. Such systems are required for machine vision from moving platforms such as robots, intelligent vehicles, and unattended large field of regard cameras with a small field of view. Our approach is based on the use of a deformable shape model for humans coupled with a novel variant of the Condensation algorithm that uses quasi-random sampling for efficiency. This allows the use of simple motion models which results in algorithm robustness, enabling us to handle unknown camera /human motion with unrestricted camera viewing angles. We present the details of our human tracking algorithms and some examples from pedestrian tracking and automated surveillance.

We describe the use of flexible models for representing the shape and grey-level appearance of human faces. These models are controlled by a small number of parameters which can be used to code the overall appearance of faces for classification purposes. The model parameters control both inter-class and within-class variation. Discriminant analysis techniques are employed to enhance the effect of those affecting inter-class variation, which are useful for classification. We show that with this coding scheme, good recognition results can be obtained, even when viewpoint, illumination and facial expression are allowed to change.

The task of fitting parametric curve models to the boundaries of perceptually meaningful image regions is a key problem in computer vision with numerous applications, such as image segmentation, pose estimation, object tracking, and 3-D reconstruction. In this article, we propose the Contracting Curve Density (CCD) algorithm as a solution to the curve-fitting problem.

A new algorithm is described for refining the pose of a model of a rigid object, to conform more accurately to the image structure. Elemental 3D forces are considered to act on the model. These are derived from directional derivatives of the image local to the projected model features. The convergence properties of the algorithm is investigated and compared to a previous technique. Its use in a video sequence of a cluttered outdoor traffic scene is also illustrated and assessed. 1 Introduction We report a new approach to the problem of recovering an accurate estimate of the pose of a rigid object, given an initial coarse estimate. The method uses an "active" model of the object, the pose of which is successively updated according to "forces" derived by examining local peaks of directional derivatives of the image, under the control of the current estimate. Unlike previous methods using active models [5, 8, 9, 16], we derive a set of elemental forces acting in 3D (rather than in the 2D...

Abstract — Medical Image Analysis plays a crucial role in the diagnosis, planning, control and follow-up of therapy. To be combined efficiently with medical robotics, Medical Image Analysis can be supported by the development of specific computational models of the human body operating at various levels. We describe in this article a hierarchy of these computational models, including the geometrical, physical and physiological levels, and illustrate their potential use in a number of advanced medical applications including image guided, robot-assisted and simulated medical interventions. We conclude this article with scientific perspectives.