We present two real-time hidden Markov model-based systems for recognizing sentence-level continuous American Sign Language (ASL) using a single camera to track the user’s unadorned hands. The first system observes the user from a desk mounted camera and achieves 92 percent word accuracy. The second system mounts the camera in a cap worn by the user and achieves 98 percent accuracy (97 percent with an unrestricted grammar). Both experiments use a 40-word lexicon.

Using hidden Markov models (HMM's), an unobstrusive single view camera system is developed that can recognize hand gestures, namely, a subset of American Sign Language (ASL). Previous systems have concentrated on finger spelling or isolated word recognition, often using tethered electronic gloves for input. We achieve high recognition rates for full sentence ASL using only visual cues. A forty word lexicon consisting of personal pronouns, verbs, nouns, and adjectives is used to create 494 randomly constructed five word sentences that are signed by the subject to the computer. The data is separated into a 395 sentence training set and an independent 99 sentence test set. While signing, the 2D position, orientation, and eccentricity of bounding ellipses of the hands are tracked in real time with the assistance of solidly colored gloves. Simultaneous recognition and segmentation of the resultant stream of feature vectors occurs five times faster than real time on an HP 735. With a strong ...

We describe our research toward building systems that include a complex, multi-state model of human dynamic behavior. This can allow us to predict human behavior over short periods of time, in order to create control systems that intelligently complement the human's action. To accomplish this requires inferring the internal state of the human, and then correctly adapting the remainder of the system to achieve optimal performance. We describe methods for achieving this goal, and report an initial experiment in which we were able to achieve 95% accuracy at predicting automobile driver's actions from their initial preparatory movements. 1 Introduction Our approach is to modeling human behavior is to consider the human as a Markov device with a (possibly large) number of internal `mental' states, each with its own particular control behavior, and inter-state transition probabilities (e.g., in a car the states might be passing, following, turning, etc.). A simple example of this type of h...

Modern wearable computer designs package workstation level performance in systems small enough to be worn as clothing. These machines enable technology to be brought where it is needed the most for the handicapped: everyday mobile environments. This paper de- scribes a research effort to make a wearable computer that can recognize (with the possible goal of translat- ing) sentence level American Sign Language (ASL) using only a baseball cap mounted camera for input. Current accuracy exceeds 97% per word on a 40 word lexicon.

Abstract. The paper presents a technique to automatically track the progress of maintenance or assembly tasks using body worn sensors. The technique is based on a novel way of combining data from accelerometers with simple frequency matching sound classification. This includes the intensity analysis of signals from microphones at different body locations to correlate environmental sounds with user activity. To evaluate our method we apply it to activities in a wood shop. On a simulated assembly task our system can successfully segment and identify most shop activities in a continuous data stream with zero false positives and 84.4 % accuracy. 1

Handwriting is a common, natural form of communication for humans, and therefore it is useful to utilize this modality as a means of input to machines. One well known method of classifying individual characters or words is template matching. We demonstrate a template-based system for online character recognition where the number of representative templates is determined automatically. These templates can be viewed as representing dierent styles of writing a particular character. The templates are then used as a reference for ecient classication using decision trees. Overall, our classier achieves an 86.9% accuracy on a set of 17,928 alphanumeric characters (36 classes; 10 digits and 26 lowercase letters) with a throughput of over 8 characters per second on a 296MHz Sun UltraSparc. | Key words: Clustering, string matching, online handwriting, prototypes, decision trees 1

A writer independent handwriting recognition system must be able to recognize a wide variety of handwriting styles, while attempting to obtain a high degree of accuracy when recognizing data from any one of those styles. As the number of writing styles increases, so does the variability of the data's distribution. We then have an optimization problem: how to best model the data, while keeping the representation as simple as possible? If we can identify N different styles of writing individual characters (referred to as lexemes), these can then be modeled as N relatively simple independent distributions. We describe here a template-based system using a string-matching distance measure for the recognition of on-line handwriting which takes advantage of lexemes to reduce the number of templates that must be stored. A method of identifying lexemes and lexeme representatives is shown. Experimental results are given for a set of 600 digits taken from 21 different writers and an...

In order to provide relevant information to mobile users, such as workers engaging in the manual tasks of maintenance and assembly, a wearable computer requires information about the user’s specific activities. This work focuses on the recognition of activities that are characterized by a hand motion and an accompanying sound. Suitable activities can be found in assembly and maintenance work. Here, we provide an initial exploration into the problem domain of continuous activity recognition using on-body sensing. We use a mock “wood workshop ” assembly task to ground our investigation. We describe a method for the continuous recognition of activities (sawing, ham-mering, filing, drilling, grinding, sanding, opening a drawer, tightening a vise, and turning a screwdriver) using microphones and 3-axis accelerometers mounted at two positions on the user’s arms. Potentially “interesting ” activities are segmented from continuous streams of data using an analysis of the sound intensity detected at the two different locations. Activity classification is then performed on these detected segments using linear discriminant analysis (LDA) on the sound channel and hidden Markov

ion Signal Level Semantic Level Figure 13: Multimodal Design Space (adapted from [224]) system in the design space is the pivotal center of its features. According to the characterization of an interaction along the two dimensions, fusion, and use of modalities, four basic types of multimodal interactions can be distinguished: alternative, synergistic, exclusive, and concurrent multimodal interaction, as shown in Figure 13. Obviously, synergistic systems subsume the other three classes of multimodal systems. Therefore, architectural models of multimodal integration (as presented in the next subsection and in Section 9) are sufficient if they are able to model synergistic cooperation of modalities. 6.2.2 Fusion of Multimodal Input Fusion of multimodal input events can occur on different levels, ranging from signal-level to semantic-level. Signal-level fusion (or lexical fusion [224]) performs the combination of multimodal input at the level of the input signal. Signal-level fusion has...

this document. The first section, in chapter 3, develops a model for syntactic dependencies based on word-category n-grams. The second section, in chapter 4, extends this model by allowing short-range word relations to be captured through the incorporation of selected word n-grams.