. The appearance of an object is composed of local structure. This local structure can be described and characterized by a vector of local features measured by local operators such as Gaussian derivatives or Gabor filters. This article presents a technique where appearances of objects are represented by the joint statistics of such local neighborhood operators. As such, this represents a new class of appearance based techniques for computer vision. Based on joint statistics, the paper develops techniques for the identification of multiple objects at arbitrary positions and orientations in a cluttered scene. Experiments show that these techniques can identify over 100 objects in the presence of major occlusions. Most remarkably, the techniques have low complexity and therefore run in real-time. 1. Introduction The paper proposes a framework for the statistical representation of the appearance of arbitrary 3D objects. This representation consists of a probability density function or jo...

This paper demonstrates a new application of computer vision to digital libraries -- the use of texture for annotation, the description of content. Vision-based annotation assists the user in attaching descriptions to large sets of images and video. If a user labels a piece of an image as "water," a texture model can be used to propagate this label to other "visually similar" regions. However, a serious problem is that no single model has been found to be good enough to reliably match human perception of similarity in pictures. Rather than using one model, the system described here knows several texture models, and is equipped with the ability to choose the one which "best explains" the regions selected by the user for annotating. If none of these models suffices, then it creates new explanations by combining models. Examples are given of annotations propagated by the system on natural scenes. The system provides an average gain of four to one in label prediction over a set of 98 image...

In this paper we present Perseus, a purposive visual system used by our robot, CHIP, to locate objects being pointed at by people. Perseus uses knowledge about the task and environment at all levels of processing to more accurately and efficiently perform visual tasks. 1 Introduction One of the tasks our robot, CHIP, performs quite well is to pick up trash around our offices and throw it away[6]. Sometimes we get tired of watching it clean up the entire room and decide we want it to pick up one particular piece of trash. Providing a verbal description of the trash's location to CHIP is awkward; it is far more natural to simply point at it. For CHIP to find objects this way it needs to notice people, recognize when they are pointing, determine which area they are pointing to, and find the object in that area. In this paper we present Perseus, a visual architecture implemented for CHIP, that enables it to perform this task. One important aspect of Perseus is that processing at all leve...

Generalised color moments combine shape and color information and put them on an equal footing. Rational expressions of such moments can be designed, that are invariant under both geometric deformations and photometric changes. These generalised color moment invariants are e#ective features for recognition under changing viewpoint and illumination. The paper gives a systematic overview of such moment invariants for several combinations of deformations and photometric changes. Their validity and potential is corroborated through a series of experiments. Both the cases of indoor and outdoor images are considered, as illumination changes tend to di#er between these circumstances. Although the generalised color moment invariants are extracted from planar surface patches, it is argued that invariant neighbourhoods o#er a concept through which they can also be used to deal with 3D objects and scenes.

This paper contributes to the viewpoint and illumination independent recognition of planar color patterns such as labels, logos, signs, pictograms, etc. by means of moment invariants. It introduces the idea of using powers of the intensities in the different color bands of a color image and combinations thereof for the construction of the moments. First, a complete classification is made of all functions of such moments which are invariant under both affine deformations of the pattern (thus achieving viewpoint invariance) as well as linear changes of the intensity values in the individual color bands (hence, coping with changes in the irradiance pattern due to different lighting conditions and/or viewpoints). The discriminant power and classification performance of these new invariants for color pattern recognition has been tested on a data set consisting of images of real outdoors advertising panels. Furthermore, a comparison to moment invariants presented in literature ([1] and [2]) ...

We recently formulated a new approach for computing invariant features from infrared (IR) images. That approach is unique in the field since it considers not just surface reflection and surface geometry in the specification of invariant features, but it also takes into account internal object composition and thermal state which affect images sensed in the non-visible spectrum. In this paper we extend the thermophysical algebraic invariance (TAI) formulation for the interpretation of uncalibrated infrared imagery, and further reduce the information that is required to be known about the environment. Features are defined such that they are functions of only the thermophysical properties of the imaged objects. In addition, we show that the distribution of the TAI features can be accurately modeled by symmetric alpha-stable models. This approach is shown to yield robust classifier performance. Results on ground truth data and real infrared imagery are presented. The application of this sch...

Interpersonal communication involves more than simply spoken information. Gestures arecommonly used to more e#ciently and precisely communicate. An important gesture because of its descriptive power and frequency of use is pointing. Toproduce a more natural and powerful human-robot interface, we have developed a purposive visual architecturecalled Perseus and have usedittolocate objectsaperson is pointing to. With Perseus, in real-time, we are able to determine when a person enters the scene, track the relevant parts of the person including the hands and head, and recognize when she is pointing. Once the person points, the object pointedtoislocated. The Perseus architecture allows knowledge about the task and context to be used at all levels of visual analysis for improvedperformance. This knowledge is explicitly represented in the Perseus system to facilitate the extension of Perseus to other tasks and environments. In this paper we describe Perseus and how it is used to so...

An important issue in developing a model-based vision system is the specification of features that are - (a) invariant to viewing and scene conditions, and also - (b) specific, i.e., the feature must have different values for different classes of objects. We formulate a new approach for establishing invariant features. Our approach is unique in the field since it considers not just surface reflection and surface geometry in the specification of invariant features, but it also takes into account internal object composition and state which affect images sensed in the non-visible spectrum. A new type of invariance called Thermophysical Invariance is defined. Features are defined such that they are functions of only the thermophysical properties of the imaged objects. The approach is based on a physics-based model that is derived from the principle of the conservation of energy applied at the surface of the imaged object. This research was supported by the AFOSR contract F49620-93-C-0063...

An important issue in developing a Model-Based Vision approach is the specification of features that are - (a) invariant to viewing and scene conditions, and also - (b) specific, i.e., the feature must have different val- ues for different classes of objects. We formulate a new approach for establishing invariant features. Our approach is unique in the field since it considers not just surface reflection and surface geometry in the specification of invariant features, but it also takes into account internal object composition and state which affect images sensed in the non-visible spectrum. A new type of invariance called Thermophysical Invariance is defined. Features are defined such that they are functions of only the thermophysical properties of the imaged objects. The approach is based on a physicsbased model that is derived from the principle of the conservation of energy applied at the surface of the imaged object.

"Stuff Analysis" is a term used to denote the analysis of texture, or collective qualities, in an image rather than the "things" which might make up an image. The stuff, in this case, are the summed pixel responses to the biologically-inspired filtering of each colour plane and level in a sub-sampled and enhanced image pyramid. This implementation skips processing intensive object segmentation and infers high-level categories from low-level histogram analysis of the directional and colour components. It uses a weight-less feed-forward RAM net to interpret a binary representation of the filter responses. Correct block categorisation has been shown to occur in images not `seen' before, across a wide range of subject material. Keywords: Biologically inspired, n-tuple, spreading, filter, histogram analysis, Wisard, discriminator, image segmentation, stuff analysis, image pyramid. 1 Introduction Multi-media libraries are becoming increasingly available and growing rapidly in size as digita...