We report research performed on gesture analysis and mapping to music. Various movements were recorded using 3D optical motion capture. Using this system, we produced animations from movements/dance, and generate in parallel the soundtrack from the dancer's movements. Prior to the actual sound mapping process, we performed various motion analyses. We present here two methods, both independent of specific orientation or location of the subject. The first deals with gestural segmentation, while the second uses pattern recognition.

We present a multimedia project incorporating music and dance. We used a 3D motion capture system to produce animations from dance and generate the soundtrack from the dancer's movements. Movement analysis is performed to extract the important features of a particular gesture. Based on the parameters chosen from this analysis, various mappings between gesture and music are applied. In particular, the motion capture data is used to trigger and modify the timbre of sounds. This paper describes the method and the interactive environment that are under development.

This paper describes a generalized motion-based framework for the generation of large musical control fields from imaging data. The framework is general in the sense that it does not depend on a particular source of sensing data. Real-time images of stage performers, pre-recorded and live video, as well as more exotic data from imaging systems such as thermography, pressure sensor arrays, etc. can be used as a source of control. Feature points are extracted from the candidate images, from which motion vector fields are calculated. After some processing, these motion vectors are mapped individually to sound synthesis parameters. Suitable synthesis techniques include granular and microsonic algorithms, additive synthesis and micro-polyphonic orchestration. Implementation details of this framework is discussed, as well as suitable creative and artistic uses and approaches.