The ability to recognize humans and their activities by vision is key for a machine to interact intelligently and effortlessly with a human-inhabited environment. Because of many potentially important applications, “looking at people ” is currently one of the most active application domains in computer vision. This survey identifies a number of promising applications and provides an overview of recent developments in this domain. The scope of this survey is limited to work on whole-body or hand motion; it does not include work on human faces. The emphasis is on discussing the various methodologies; they are grouped in 2-D approaches with or without explicit shape models and 3-D approaches. Where appropriate, systems are reviewed. We conclude with some thoughts about future directions. c ○ 1999 Academic Press 1.

w4 is a real time visual surveillance system for detecting and tracking multiple people and monitoring their activities in an outdoor environment. It operates on monocular gray-scale video imagery, or on video imagery from an infrared camera. W4 employs a combination of shape analysis and tracking to locate people and their parts (head, hands, feet, torso) and to create models of people's appearance so that they can be tracked through interactions such as occlusions. It can determine whether a foreground region contains multiple people and can segment the region into its constituent people and track them. W4 can also determine whether people are carrying objects, and can segment objects from their silhouettes, and construct appearance models for them so they can be identified in subsequent frames. W4 can recognize events between people and objects, such as depositing an object, exchanging bags, or removing an object. It runs at 25 Hz for 320x240 resolution images on a 400 Mhz dual-Pentium II PC.

ras) moving? but, rather What is happening? Unfortunately, this new labeling problem is not as welldefined as the previously addressed questions of geometry. Bobick [6] considers the range of motion interpretation problems and proposes a taxonomy of approaches. At the top and intermediate levelsaction and activity, respectively are situations in which knowledge other than the immediate motion is required to generate the appropriate label. The most primitive level, however, is movementa motion whose execution is consistent and easily characterized by a definite space-time trajectory in some feature space. Such consistency of execution implies that for a given viewing condition there is consistency of appearance. Put simply, movements can be described by their appearance. This paper presents a novel, appearance-based approach to the recognition of human movement. Our work stands in contrast to many recent efforts to recover the full threedimensional reconstruction of th

A comprehensive survey of computer vision-based human motion capture literature from the past two decades is presented. The focus is on a general overview based on a taxonomy of system functionalities, broken down into four processes: initialization, tracking, pose estimation, and recognition. Each process is discussed and divided into subprocesses and/or categories of methods to provide a reference to describe and compare the more than 130 publications covered by the survey. References are included throughout the paper to exemplify important issues and their relations to the various methods. A number of general assumptions used in this research field are identified and the character of these assumptions indicates that the research field is still in an early stage of development. To evaluate the state of the art, the major application areas are identified and performances are analyzed in light of the methods

Human motion analysis is receiving increasing at-tention from computer vision researchers. This inter-est is motivated by a wide spectrum of applications, such as athletic performance analysis, surveillance, man-machine interfaces, content-based image storage and retrieval, and video conferencing. This paper gives an overview of the various tasks involved in motion analysis of the human body. We focus on three major areas related to interpreting human motion: 1) motion analysis involving human body parts, 2) tracking of human motion wing single or multiple cameras, and 8) recognizing human activities from image sequences. Motion analysis of human body parts involves the low-level segmentation of the human body into segments connected by joints, and recovers the 3D structure of the human body using its 20 projections over a se-quence of images. Ilfacking human motion wing a single or multiple cameras focuses on higher-level pro-cessing, in which moving humans are tracked without identifying specific parts of the body structure. After successfully matching the moving human image)?om one frame to another in image sequences, understand-ing the human movements or activities comes natu-rally, which leads to our discussion of recognizing hu-man activities. The review is illustrated by ezamples. 1

A new view-based approach to the representation and recognition of action is presented. The basis of the representation is a temporal template --- a static vector-image where the vector value at each point is a function of the motion properties at the corresponding spatial location in an image sequence. Using 18 aerobics exercises as a test domain, we explore the representational power of a simple, two component version of the templates: the #rst value is a binary value indicating the presence of motion, and the second value is a function of the recency of motion in a sequence. We then develop a recognition method which matches these temporal templates against stored instances of views of known actions. The method automatically performs temporal segmentation, is invariant to linear changes in speed, and runs in real-time on a standard platform. We recently incorporated this technique into the KidsRoom: an interactive, narrative play-space for children. 1 Introduction The recent shift...

. This paper demonstrates a content-based retrieval strategy that can tell whether there are naked people present in an image. No manual intervention is required. The approach combines color and texture properties to obtain an effective mask for skin regions. The skin mask is shown to be effective for a wide range of shades and colors of skin. These skin regions are then fed to a specialized grouper, which attempts to group a human figure using geometric constraints on human structure. This approach introduces a new view of object recognition, where an object model is an organized collection of grouping hints obtained from a combination of constraints on geometric properties such as the structure of individual parts, and the relationships between parts, and constraints on color and texture. The system is demonstrated to have 60% precision and 52% recall on a test set of 138 uncontrolled images of naked people, mostly obtained from the internet, and 1401 assorted control images, drawn f...

A new method for representing and recognizing human bodymovements is presented. Assuming the availability of Cartesian tracking data, we develop techniques for representation of movements basedon spacecurves in subspaces of a "phase space." The phase space has axes of joint angles and torso location and attitude, and the axes of the subspaces are subsets of the axes of the phase space. Using this representation we develop a system for learning new movements from ground truth data by searching for constraints which are in effect during the movement to be learned, and not in effect during other movements. We then use the learned representation for recognizing movements in data. Prior approaches by other researchers used a small number of classification categories, which demanded less attention to representation. We train and test the system on nine fundamental movements from classical ballet performed by two dancers. The system learns and accurately recognizes the nine movements in an un...

We present a new method for the 3D model-based tracking of human body parts. To mitigate the difficulties arising due to occlusion among body parts, we employ multiple calibrated cameras in a mutually orthogonal configuration. In addition, we develop criteria for a time varying active selection of a set of cameras to track the motion of a particular human part. In particular, at every frame, each camera tracks a number of parts depending on the visibility of these parts and the observability of their predicted motion from the specific camera. To relate points on the occluding contours of the parts to points on their models we apply concepts from projective geometry. Then, within the physics-based framework we compute the generalized forces applied from the parts' occluding contours to model points of the body parts. These forces update the translational and rotational degrees of freedom of the model, such as to minimize the discrepancy between the sensory data and the estimated model s...

Motion perception and interpretation plays an important role in the human visual system. It helps us recognize different objects and their motion in a scene, infer their relative depth, their rigidity, etc. In psychology, this process has been studied extensively by Johansson using moving light displays (MLDs). MLDs consist of bright spots attached to the joints of an actor dressed in black, and moving in front of a dark background. The collection of spots carry only 2D information and no structural information, since they are not connected. A set of static spots remained meaningless to observers, while their relative movement created a vivid impression of a person walking, running, dancing, etc. The gender of a person, and even the gait of a friend can be recognized based solely on the motion of those spots. There are two theories about the interpretation of MLD type stimuli, from a psychology point of view. In the first, people use motion information in the MLD to recover t...