We report research performed on gesture analysis and mapping to music. Various movements were recorded using 3D optical motion capture. Using this system, we produced animations from movements/dance, and generate in parallel the soundtrack from the dancer's movements. Prior to the actual sound mapping process, we performed various motion analyses. We present here two methods, both independent of specific orientation or location of the subject. The first deals with gestural segmentation, while the second uses pattern recognition.

In this paper we present a new segmentation solution for extracting motion patterns from motion capture data by searching for critical keyposes in the motion sequence. A rank is established for critical keyposes that identifies the significance of the directional change in motion data. The method is based on entropy metrics, specifically the mutual information measure. Displacement histograms between frames are evaluated and the mutual information metric is employed in order to calculate the inter-frame dependency. The most significant keypose identifies the largest directional change in the motion data. This will have the lowest mutual information level from all the candidate keyposes. Less significant keyposes are then listed with higher mutual information levels. The results show that the method has higher sensitivity in the directional change than methods based on the magnitude of the velocity alone. This method is intended to provide a summary of a motion clip by ranked keyposes, which is highly useful in motion browsing and motion retrieve database system.

This paper discusses a novel approach for capturing and translating isolated gestures of American Sign Language into spoken and written words. The instrumented part of the system combines an AcceleGlove and a two-link arm skeleton. Gestures of the American Sign Language are broken down into unique sequences of phonemes called Poses and Movements, recognized by software modules trained and tested independently on volunteers with different hand sizes and signing ability. Recognition rates of independent modules reached up to 100 % for 42 postures, 6 orientations, 11 locations and 7 movements using linear classification. The overall sign recognizer was tested using a subset of the American Sign Language dictionary comprised by 30 one-handed signs, achieving 98 % accuracy. The system proved to be scalable: when the lexicon was extended to 176 signs and tested without retraining, the accuracy was 95%. This represents an improvement over classification based on Hidden Markov Models and Neural Network.