We present two real-time hidden Markov model-based systems for recognizing sentence-level continuous American Sign Language (ASL) using a single camera to track the user’s unadorned hands. The first system observes the user from a desk mounted camera and achieves 92 percent word accuracy. The second system mounts the camera in a cap worn by the user and achieves 98 percent accuracy (97 percent with an unrestricted grammar). Both experiments use a 40-word lexicon.

Using hidden Markov models (HMM's), an unobstrusive single view camera system is developed that can recognize hand gestures, namely, a subset of American Sign Language (ASL). Previous systems have concentrated on finger spelling or isolated word recognition, often using tethered electronic gloves for input. We achieve high recognition rates for full sentence ASL using only visual cues. A forty word lexicon consisting of personal pronouns, verbs, nouns, and adjectives is used to create 494 randomly constructed five word sentences that are signed by the subject to the computer. The data is separated into a 395 sentence training set and an independent 99 sentence test set. While signing, the 2D position, orientation, and eccentricity of bounding ellipses of the hands are tracked in real time with the assistance of solidly colored gloves. Simultaneous recognition and segmentation of the resultant stream of feature vectors occurs five times faster than real time on an HP 735. With a strong ...

Modern wearable computer designs package workstation level performance in systems small enough to be worn as clothing. These machines enable technology to be brought where it is needed the most for the handicapped: everyday mobile environments. This paper de- scribes a research effort to make a wearable computer that can recognize (with the possible goal of translat- ing) sentence level American Sign Language (ASL) using only a baseball cap mounted camera for input. Current accuracy exceeds 97% per word on a 40 word lexicon.

This work presents a design for a human computer interface capable of recognizing 25 gestures from the international hand alphabet in real-time. Principal Component Analysis (PCA) is used to extract features from images of gestures. The features represent gesture images in terms of an optimal coordinate system, in which the classes of gestures make up clusters. The system is divided into two parts: an off-line and an on-line part.

We build upon a constrained, lab-based Sign Language recognition system with the goal of making it a mobile assistive technology. We examine using multiple sensors for disambiguation of noisy data to improve recognition accuracy. Our experiment compares the results of training a small gesture vocabulary using noisy vision data, accelerometer data and both data sets combined.

This thesis describes a new, large-scale Hypermedia project ("Hyper-G") currently being developed at the Institute for Foundations of Information Processing and Computer Supported New Media (Head: Professor Hermann Maurer) of the Technical University of Graz. Experience gathered from modern Hypermedia systems, large-scale information systems, computer aided instruction and user interface design considerations led to a number of ideas, features, and examples of applications of Hyper-G. They were condensed, put into a logical relationship, and used to formulate a set of requirements. The requirements, additional design decisions, and a discussion of implementation-related issues are part of this thesis. Also, a new concept for the creation of real-time, interactive animation is presented. It is essentially a combination of Computer Animation and Hypermedia technologies, therefore it is called "Hyper-Animation". This concept is also the basis of some of the more advanced app...

This dissertation describes a user interface in which many tasks traditionally performed by a mouse are instead performed using visual recognition of hand gestures. The goals are to explore both how a vision system should be designed to recognize hand gestures, and how they are best used in a general purpose interface. Observed by a camera below the screen, the user manipulates objects directly with gestures incorporating both motion and pose. Task and domain knowledge provide context, allowing real-time recognition on standard PC hardware. A color-based algorithm is trained to segment user's hands from complex backgrounds without visual aids. Training uses a novel combination of both positive and negative data to improve segmentation quality. The apparent path of the hand is smoothed with an algorithm which reduces the types of noise inherent in the domain but leaves a cursor motion on the screen that feels natural for the user. Salient features of the motion are extracted, including a newly discovered natural gesture (a “Comma”), which helps provide punctuation for each gestural sentence. Neural networks are trained to classify the pose of the user's hand from cropped and preprocessed images. The nets correctly classify 90-95 % of the hand images in real time. A transition network encodes the interaction language. It controls the application of feature extraction operators and interprets their results to determine when to perform actions on the user's behalf. The style of interaction is based on studies of natural gesticulation and incorporates various features designed to make it natural and easy for the user to remember. The system demonstrates a 80-90 % success rate on most tasks. Object selection time for large objects is demonstrated to be equal or superior to that of a mouse. Object selection performance is modeled accurately by augmenting Fitts ' Law with terms for lag and random cursor noise. Finally, the suitability of gesture for this type of task is considered. Various interaction styles are examined, and problems specific to hand gesture are discussed.

Recent approaches to providing users with a more natural method of interacting with computer applications have shown that more than one mode of input can be both beneficial and intuitive as a communication medium between humans and computers. Two modalities in particular, whole-hand and speech input, represent a natural form of communication that has been ingrained in our physical and mental makeup since birth. In this thesis, we investigate the use of whole-hand and speech input in virtual environments in the context of two applications domains: scientific visualization and interior design. By examining the two modalities individually and in combination, and through the creation of two application prototypes (Multimodal Scientific Visualization Tool and Room Designer), we present anumber of contributions including a set of interface guidelines and interaction techniques for whole-hand and speech input.

We present FreeDigiter, an interface for mobile devices which enables rapid entry of digits using finger gestures. FreeDigiter senses via an infrared proximity sensor and a dual axis accelerometer and requires little signal processing. Initial experiments attain accuracy rates of 99.0%, and the system is tolerant to highly varying lighting conditions. The FreeDigiter system requires little power and could be implemented in a very small form factor appropriate for in– ear hearing aids, small MP3 players, and hands–free mobile phone headsets. 1

: Previous work in recognition of hand gestures has concentrated on classification of hand shapes, with relatively little work done on hand motions. This paper describes a recurrent neural network which has been trained to classify sixteen different hand trajectories, including relatively complex paths such as circles and backand -forth motions. The network's ability to anticipate the classification of an incomplete gesture is also examined, and its implications for segmentation of gestures is discussed. Introduction Computer recognition of human hand gestures has potential for application in many fields such as virtual reality interfaces, robotic control and automated sign language translation. Hand data can be captured either via a camera and image processing techniques, or directly through an instrumented glove worn by the user. Pattern recognition techniques can then be applied to this data to classify the gesture made by the user. Components of hand gestures Most of the analys...