We describe an inertial gesture recognition framework composed of three parts. The first is a compact, wireless six-axis inertial measurement unit to fully capture three-dimensional motion. The second, a gesture recognition algorithm, analyzes the data and categorizes it on an axis-by-axis basis as simple motions (straight line, twist, etc.) with magnitude and duration. The third allows an application designer to combine recognized gestures both concurrently and consecutively to create specific composite gestures can then be set to trigger output routines.

This paper presents a system to control the playback of audio files by means of the standard classical conducting technique. Computer vision techniques are developed to track a conductor's baton, and the gesture is subsequently analysed. Audio parameters are extracted from the sound-file and are further processed for audio beat tracking. The sound-file playback speed is adjusted in order to bring the audio beat points into alignment with the gesture beat points. The complete system forms all parts necessary to simulate an orchestra reacting to a conductor 's baton.

This paper describes three different types of real-time optical tracking systems developed at the MIT Media Laboratory for use as expressive human-computer interfaces in music, dance, and interactive multimedia performances. Two of these, a multimodal conducing baton and a scanning laser rangefinder, are essentially hardware-based, while the third is a computer vision system that can identify and track different segments of the performers body. We discuss the technical concepts behind these devices and outline their applications in music and dance environments. 1.

Digital musical instruments do not depend on physical constraints faced by their acoustic counterparts, such as characteristics of tubes, membranes, strings, etc. This fact permits a huge diversity of possibilities regarding sound production, but on the other hand strategies to design and perform these new instruments need to be devised in order to provide the same level of control subtlety available in acoustic instruments. In this paper I review various topics related to gestural control of music using digital musical instruments and identify possible trends in this domain.

Inertial measurement components, which sense either acceleration or angular rate, are being embedded into common user interface devices more frequently as their cost continues to drop dramatically. These devices hold a number of advantages over other sensing technologies: they measure relevant parameters for human interfaces and can easily be embedded into wireless, mobile platforms. The work in this dissertation demonstrates that inertial measurement can be used to acquire rich data about human gestures, that we can derive efficient algorithms for using this data in gesture recognition, and that the concept of a parameterized atomic gesture recognition has merit. Further we show that a framework combining these three levels of description can be easily used by designers to create robust applications.

This paper describes the development of a musical interface based on electromagnetic tagging technology, where an ensemble of passively tagged objects is identified and tracked in real time when placed in the vicinity of a reader. As the system is able to identify and update the state of many (30 or more) tags simultaneously, they can be used together in any combination -- e.g., several can sit on a surface at fixed distances from the reader while others can be handheld or worn by a single user or multiple performers. This interface is able to detect both free gesture (position and orientation of the objects) as well as local or tactile variables (e.g., pressure). We describe a series of controllers that exploit the musical possibilities of this architecture -- the somewhat constrained Musical Trinkets, where objects were tied to simple notes and musical effects, and its successor, the Musical Navigatrics, which enabled dynamic overdubbing and control of complex musical sequences and sonic textures. We close with a description of a very simple and inexpensive actively-tagged tracking system capable of much wider range.

Although a number of conducting gesture analysis and following systems have been developed over the years, most of the projects either primarily concentrated on tracking tempo and amplitude indicating gestures while not taking expressive gestures into account, or implemented individual mapping techniques for expressive gestures that varied from research to research. There is a clear need for a uniform process that could be applied toward analysis of both indicative and expressive gestures. The conducting gesture recognition system is implemented on the basis of Hidden Markov Model (HMM) process. An external HMM object is developed for Max/MSP software. Training and recognition procedures are applied toward both right hand beat- and amplitude- indicative gestures, and left hand expressive gestures. Continuous recognition of right-hand gestures is incorporated into a real-time gesture analysis and performance system in Max/MSP/Jitter environment. 1

This paper describes a proof-of-principle demonstration of a novel and simple active tracking mechanism using a laser diode, steering mirrors, and a single non-imaging photodetector. Tracking is based on the analysis of a temporal signal corresponding to the amount of backscattered light produced during a rapid, local circular scan (or saccade) around the presumed object position. The simplicity of the system is such that, using state-of-the-art Micro-Opto-Electro-Mechanical-System (MOEMS) technology, it would be possible to integrate the whole system on a chip, making it an interesting input interface for portable computing devices.

A system to track an uninstrumented conductor's baton by computer vision techniques is developed. An initial seek mode locates a baton in an image frame without prior knowledge; a subsequent track mode uses temporal information to cope with motion blur and noise. The seek mode utilises the characteristic form of detected edges to guide a maximal intensity trace. The track mode enhances this technique by dovetailing with optical flow to give robust tracking, even with considerable motion blur and poor quality images.

We present a prototype of a new musical interface for Japanese drumming techniques and styles. Our design used in the Aobachi drumming sticks provides 5 gesture parameters (3 axes of acceleration, and 2 axes of angular velocity) for each of the two sticks and transmits this data wirelessly using Bluetooth technology. This system utilizes minimal hardware embedded in the two drumming sticks, allowing for gesture tracking of drum strokes by an interface of traditional form, appearance, and feel. Aobachi is portable, versatile, and robust, and may be used for a variety of musical applications, as well as analytical studies.