Abstract: Traditional musical instruments provide compelling metaphors for human-computer interfacing, both in terms of input (physical, gestural performance activities) and output (sound diffusion). The violin, one of the most refined and expressive of traditional instruments, combines a peculiar physical interface with a rich acoustic diffuser. We have built a new instrument that includes elements of both the violin's physical performance interface and its spatial filtering audio diffuser, yet eliminates both the resonating body and the strings. The instrument, BoSSA (Bowed-Sensor-Speaker-Array), is an amalgamation and extension of our previous work with violin interfaces, physical models, and directional tonal radiation studies. In addition to describing the various physical and software elements that make up BoSSA, we discuss some of its musical features and potentials; we are particularly impressed by the sense of presence and intimacy it provides, and by its potential for creating a new kind of electronic chamber music. Traditional musical instruments provide compelling metaphors for human-computer interfacing,

As an outgrowth of our interest in dense wireless sensing and expressive applications of wearable computing, we have developed the world's most versatile human-computer interface for the foot. By dense wireless sensing, we mean the remote acquisition of many different parameters with a compact, autonomous sensor cluster. We have developed such a low-power sensor card to measure over 16 continuous quantities and transmit them wirelessly to a remote base station, updating all variables at 50 Hz. We have integrated a pair of these devices onto the feet of dancers and athletes, measuring continuous pressure at 3 points near the toe, dynamic pressure at the heel, bidirectional bend of the sole, height of each foot off conducting strips in the stage, angular rate of each foot about the vertical, angular position of each foot about the Earth's local magnetic field, as well as their tilt and low-G acceleration, 3-axis shock acceleration (from kicks and jumps), and position (via an integrated s...

In this paper, the design and construction of a new violin interface, the Hyperbow, is discussed. The motivation driving the research of this instrument was the desire to create a violin bow capable of measuring the most intricate aspects of violin technique--the subtle elements of physical gesture that immediately and directly impact the sound of the instrument while playing. In order to provide this insight into the subtleties of bow articulation, a sensing system has been integrated into a commercial carbon fiber bow to measure changes in position, acceleration, and the downward and lateral strains of the bow stick. The sensors were fashioned using an electromagnetic field sensing technique, commercial MEMS accelerometers, and foil strain gauges. The measurement techniques used in this work were found to be quite sensitive and yielded sensors that were easily controllable by a player using traditional right hand bowing technique.

This paper describes three different types of real-time optical tracking systems developed at the MIT Media Laboratory for use as expressive human-computer interfaces in music, dance, and interactive multimedia performances. Two of these, a multimodal conducing baton and a scanning laser rangefinder, are essentially hardware-based, while the third is a computer vision system that can identify and track different segments of the performers body. We discuss the technical concepts behind these devices and outline their applications in music and dance environments. 1.

This paper provides a review of gestural control of sound synthesis in the context of the design and evaluation of digital musical instruments. It discusses research in various areas related to this field and equally focuses on four main topics: analysis of music performers’ gestures, gestural capture technologies, real-time sound synthesis methods, and strategies for mapping gesture variables to sound synthesis input parameters. Finally, this approach is illustrated by presenting an application of this research to the control of digital audio effects.

The physical user interface is an increasingly significant factor limiting the effectiveness of our interactions with and through technology. This thesis introduces Electric Field Imaging, a new physical channel and inference framework for machine perception of human action. Though electric field sensing is an important sensory modality for several species of fish, it has not been seriously explored as a channel for machine perception. Technological applications of field sensing, from the Theremin to the capacitive elevator button, have been limited to simple proximity detection tasks. This thesis presents a solution to the inverse problem of inferring geometrical information about the configuration and motion of the human body from electric field measurements. It also presents simple, inexpensive hardware and signal processing techniques for making the field measurements, and several new applications of electric field sensing. The signal

In this paper, we present a wireless sensor platform designed for processing multipoint human motion with low latency and high resolution. One application considered here is interactive dance, in which a choreographer wishes to translate the movements of multiple dancers into real-time audio or video content to accompany the performance. This can only be accomplished using a distributed measurement system capable of responding quickly with enough information to describe the expressive movements of multiple people. Similar requirements exist for biomechanical analysis, especially in the context of athletic training, where high resolution is demanded, and instant feedback is also desirable. Our approach to addressing such aggressive requirements involves a high-speed wireless network of compact inertial measurement units (IMUs) that can be worn at various locations on the body. Each device is equipped with its own 1Mbps radio link and a full six-axis IMU, as well as a capacitive node-to-node proximity sensor. Currently, the system supports real-time data collection and processing for up to 25 nodes with 100Hz full state updates, thereby handling much higher data rates than its predecessors. With locally buffered data, sample rates of up to 1kHz have been achieved successfully. Early results discussed here demonstrate the feasibility of our design through testing with both dancers and professional athletes.

Digital musical instruments do not depend on physical constraints faced by their acoustic counterparts, such as characteristics of tubes, membranes, strings, etc. This fact permits a huge diversity of possibilities regarding sound production, but on the other hand strategies to design and perform these new instruments need to be devised in order to provide the same level of control subtlety available in acoustic instruments. In this paper I review various topics related to gestural control of music using digital musical instruments and identify possible trends in this domain.

This paper presents a method for the acquisition of violin instrumental gesture parameters by using a commercial two-sensor 3D tracking system based on electro-magnetic field (EMF) sensing. The methodolgy described here is suitable for acquiring instrumental gesture parameters of any bowed-string instrument, and has been devised by paying attention to intrusion, flexibility, and robustness. After reviewing relevant related work in the field, we give an overview of the application context, pointing out some basic needs to be fulfilled for our research purposes. Then, we present the steps for calibrating the system, followed by details on the computation of a number of relevant instrumental gesture parameters. The use of a number of the extracted parameters to perform score-performance alignment and database automatic annotation is also outlined. Finally, we conclude stating next steps in using acquired data, along with further developments of the methodology. many instrumental gesture parameters as possible, (2) to be accurate, (3) to be easily attached to any violin/bow, and (4) to present a small effect on the instrument playability. As a first step towards our research goals, we use the acquired parameters for automatically annotating a performance database by applying a custom score-performance alignment procedure. The paper is structured as follows. First, we offer an outlook to the related work in the field, highlighting the main differences with the approach presented here. Then, we put our work into context, pointing out our research purposes, and discussing the fulfillment of our application requirements. Section 4 outlines the calibration procedure, while Section 5 gives the details of the computation of a number of instrumental gesture parameters. In Section 6, we present the score-alignment procedure devised for annotating our database. We conclude by discussing further developments of the methodology and future uses of our gathered data. 1.

This paper describes the development of a musical interface based on electromagnetic tagging technology, where an ensemble of passively tagged objects is identified and tracked in real time when placed in the vicinity of a reader. As the system is able to identify and update the state of many (30 or more) tags simultaneously, they can be used together in any combination -- e.g., several can sit on a surface at fixed distances from the reader while others can be handheld or worn by a single user or multiple performers. This interface is able to detect both free gesture (position and orientation of the objects) as well as local or tactile variables (e.g., pressure). We describe a series of controllers that exploit the musical possibilities of this architecture -- the somewhat constrained Musical Trinkets, where objects were tied to simple notes and musical effects, and its successor, the Musical Navigatrics, which enabled dynamic overdubbing and control of complex musical sequences and sonic textures. We close with a description of a very simple and inexpensive actively-tagged tracking system capable of much wider range.