The ability to recognize humans and their activities by vision is key for a machine to interact intelligently and effortlessly with a human-inhabited environment. Because of many potentially important applications, “looking at people ” is currently one of the most active application domains in computer vision. This survey identifies a number of promising applications and provides an overview of recent developments in this domain. The scope of this survey is limited to work on whole-body or hand motion; it does not include work on human faces. The emphasis is on discussing the various methodologies; they are grouped in 2-D approaches with or without explicit shape models and 3-D approaches. Where appropriate, systems are reviewed. We conclude with some thoughts about future directions. c ○ 1999 Academic Press 1.

We present two real-time hidden Markov model-based systems for recognizing sentence-level continuous American Sign Language (ASL) using a single camera to track the user’s unadorned hands. The first system observes the user from a desk mounted camera and achieves 92 percent word accuracy. The second system mounts the camera in a cap worn by the user and achieves 98 percent accuracy (97 percent with an unrestricted grammar). Both experiments use a 40-word lexicon.

Using hidden Markov models (HMM's), an unobstrusive single view camera system is developed that can recognize hand gestures, namely, a subset of American Sign Language (ASL). Previous systems have concentrated on finger spelling or isolated word recognition, often using tethered electronic gloves for input. We achieve high recognition rates for full sentence ASL using only visual cues. A forty word lexicon consisting of personal pronouns, verbs, nouns, and adjectives is used to create 494 randomly constructed five word sentences that are signed by the subject to the computer. The data is separated into a 395 sentence training set and an independent 99 sentence test set. While signing, the 2D position, orientation, and eccentricity of bounding ellipses of the hands are tracked in real time with the assistance of solidly colored gloves. Simultaneous recognition and segmentation of the resultant stream of feature vectors occurs five times faster than real time on an HP 735. With a strong ...

Biometric authentication of gait, anthropometric data, human activities and movement disorders are presented in this paper using the Continuous Human Movement Recognition (CHMR) framework introduced in Part I. A novel biometric authentication of anthropometric data is presented based on the realization that no one is average sized in as many as 10 dimensions. These body part dimensions are quantified using the CHMR body model. Gait signatures are then evaluated using motion vectors, temporally segmented by gait dynemes, and projected into a gait space for an eigengait based biometric authentication. Left-right asymmetry of gait is also evaluated using robust CHMR left-right labeling of gait strides. Accuracy of the gait signature is further enhanced by incorporating the knee-hip angle-angle relationship popular in biomechanics gait research, together with other gait parameters. These gait and anthropometric biometrics are fused to further improve accuracy. The next biometric identifies human activities which requires a robust segmentation of the many skills encompassed. For this reason, the CHMR activity model is used to identify various activities from making a coffee to using a computer. Finally, human movement disorders were evaluated by studying patients with dopa-responsive Parkinsonism and age matched normals who were video taped during several gait cycles to determine a robust metric for classifying movement disorders. The results suggest that the R. D. Green is with the Human Interface Technology Lab, University of Canterbury, Christchurch, New Zealand. He was with the School of Electrical and Information Engineering, The University of Sydney, NSW 2006, Australia, (e-mail: richard.green@canterbury.ac.nz).

Processing speeds have increased dramatically, bitmapped displays allow graphics to be rendered and updated at increasing rates, and in general computers have advanced to the point where they can assist humans in complex tasks. Yet input technologies seem to cause the major bottleneck in performing these tasks: under-utilising the available resources, and restricting the expressiveness of application use. We use our hands constantly to interact with things: pick them up, move them, transform their shape, or activate them in some way. In the same unconscious way, we gesticulate in communicating fundamental ideas: `stop', `come closer', `over there', `no', `agreed', and so on. Gestures are thus a natural and intuitive form of both interaction and communication. This report develops the motivations for gestural input and surveys current gesture recognition techniques. A recognition technique under development at TCD, as part of the GLAD-IN-ART (EP5363) project, is also introduced. Conte...

We have developed the Hand Motion Understanding (HMU) system that understands static and dynamic signs of the Australian Sign Language (Auslan). The HMU system uses a visual 3D hand tracker for motion sensing, and an adaptive fuzzy expert system for classification of the signs. This paper presents the hand tracker that extracts 3D hand configuration data with 21 degrees-of-freedom (DOFs) from a motion sequence that is captured from a single viewpoint, with the aid of a colour-coded glove. The tracker is used for the training and evaluation of the HMU system with 22 static and dynamic signs. Before training the HMU system recognised 20 signs, and after training, it recognised 21 signs. 1. Introduction Automated gesture recognition has been an active area of research in human-computer interaction applications and sign language translation systems. The gesture recognition may be performed in two stages: the motion sensing, which extracts useful motion data from the actual motion input; a...

Signed English is a manual interpretation of English using fingerspelling and signs. A prototype of the Hand Sign Translator (HST) system was developed to graphically translate English into Signed English, using two-handed animation. The HST consists of a practical interface that aims to help users learn Signed English, and the translation process where English text is transformed into a series of images that represent corresponding signs. This paper describes the translation process which involves two stages; the input environment and the animation process. The input environment consists of text analysis in order to extract corresponding kinematic data from the database, named English-Sign Dictionary (ESD). The data is then used as an input to the animation process. Firstly, the skeleton models of keyframe images and their inbetween poses are calculated. Secondly, appropriate volume models are applied in order to surround the surface of skin. Then the shapes that are suitable for pain...

The ability to recognize humans and their activities by visionisessential for future machines to interact intelligently and e ortlessly with a human-inhabited environment. Some of the more promising applications are discussed. A prototype vision system is presented for the tracking of whole-body movement using multiple cameras. 3-D body pose is recovered at each time instant based on occluding contours. The pose-recovery problem is formulated as a search problem and entails nding the pose parameters of a graphical human model whose synthesized appearance is most similar to the actual appearance of the real human in the multi-view images. Hermite deformable contours are proposed as a tool for the 2-D contour tracking problem. The main contribution of this dissertation is that it demonstrates for the rst time a set of techniques that allow accurate vision-based 3-D tracking of arbitrary whole-body movement without the use of markers.

Hand sign recognition, in general, may be divided into two stages: the motion sensing, which extracts useful movement data from the signer's motion; and the classification process, which classifies the movement data as a sign. We have developed a prototype of the Hand Sign Classification (HSC) system that classifies a series of the full degrees-of-freedom kinematic data of a hand into sign language signs. It is built as a fuzzy expert system in which the sign knowledge can be represented by high level imprecise descriptions. Applying fuzzy logic also provides the system with the ability to produce a confidence level for an output. The HSC system has an adaptive engine that trains the system to handle variations in the movement data, or to adapt to differences amongst signers. Adaptive fuzzy systems are often compared with neural networks in their adaptability, but unlike neural networks, expert knowledge can be imposed onto the system in the form of rules. Keywords: Sign L...

Articulated motion is a subset of non-rigid motion in which the object of interest is composed of several rigid components connected to each other by ball and hinge joints. The human body, many animals and insects, and machinery all exhibit such motion. This dissertation addresses the problem of vision-based tracking and analysis of this type of motion. The importance of this problem can be seen in many application domains including surveillance, traffic monitoring, entertainment, user interfaces, medicine, sports, video annotation, and image compression. This dissertation deals with two important subproblems of the general problem: whole-body tracking and motion recognition. In whole-body tracking, the body is tracked as one unit without paying attention to the details of the posture and limbs. Current solutions to this problem suffer from being too sensitive to small changes in the environment. We present a novel approach which reduces these restrictions significantly. This is achieved by separating the concepts of a blob from that of a body and by tracking each independently while maintaining a many-to-many relationship between the two. The approach makes use of the Extended Kalman Filter and outputs trajectory information in world coordinates. The method was tested by tracking pedestrians in a variety of environments and achieved real-time performance and a high degree of robustness. Motion recognition is the high level problem of classifying an action taking place in a video sequence into one of several action categories. Most of the present approaches attempt to perform three-dimensional reconstruction of the articulated shape prior to recognition, which is an inherently difficult problem made even more difficult due to the nonrigidity of the articulated object. W...