This paper provides a review of gestural control of sound synthesis in the context of the design and evaluation of digital musical instruments. It discusses research in various areas related to this field and equally focuses on four main topics: analysis of music performers’ gestures, gestural capture technologies, real-time sound synthesis methods, and strategies for mapping gesture variables to sound synthesis input parameters. Finally, this approach is illustrated by presenting an application of this research to the control of digital audio effects.

When envisaging new digital instruments, designers do not have to limit themselves to their sonic capabilities (which can be absolutely any), not even to their algorithmic power; they must be also especially careful about the instruments' conceptual capabilities, to the ways instruments impose or suggest to their players new ways of thinking, new ways of establishing relations, new ways of interacting, new ways of organizing time and textures; new ways, in short, of playing new musics. This article explores the dynamic relation that builds between the player and the instrument, introducing concepts such as efficiency, apprenticeship and learning curve It aims at constructing a framework in which the possibilities and the diversity of music instruments as well as the possibilities and the expressive freedom of human music performers could start being evaluated.

While many new interfaces for musical expression have been presented in the past, methods to evaluate these interfaces are rare. This paper presents a method and a study comparing the potential for musical expression of different string-instrument based musical interfaces. Cues for musical expression are defined based on results of research in musical expression and on methods for musical education in instrumental pedagogy. Interfaces are evaluated according to how well they are estimated to allow players making use of their existing technique for the creation of expressive music.

The central role of the face in social interaction and non-verbal communication suggest we explore facial action as a means of musical expression. This paper presents the design, implementation, and preliminary studies of a novel system utilizing face detection and optic flow algorithms to associate facial movements with sound synthesis in a topographically specific fashion. We report on our experience with various gesture-to-sound mappings and applications, and describe our preliminary experiments at musical performance using the system.

While several researchers have grappled with the problem of comparing musical devices across performance, installation, and related contexts, no methodology yet exists for producing holistic, informative visualizations for these devices. Drawing on existing research in performance interaction, human-computer interaction, and design space analysis, the authors propose a dimension space representation that can be adapted for visually displaying musical devices. This paper illustrates one possible application of the dimension space to existing performance and interaction systems, revealing its usefulness both in exposing patterns across existing musical devices and aiding in the design of new ones.

When learning a classical instrument, people often either take lessons in which an existing body of "technique" is delivered, evolved over generations of performers, or in some cases people will "teach themselves" by watching people play and listening to existing recordings. What does one do with a complex new digital instrument? In this paper I address this question drawing on my experience in learning several very di#erent types of sophisticated instruments: the Glove Talk II real-time gesture-to-speech interface, the Digital Marionette controller for virtual 3D puppets, and pianos and keyboards. As the primary user of the first two systems, I have spent hundreds of hours with Digital Marionette and Glove-Talk II, and thousands of hours with pianos and keyboards (I continue to work as a professional musician). I will identify some of the underlying principles and approaches that I have observed during my learning and playing experience common to these instruments. While typical accounts of users learning new interfaces generally focus on reporting beginner's experiences, for various practical reasons, this is fundamentally di#erent by focusing on the expert's learning experience.

The iltur system features a novel method of interaction between expert and novice musicians through a set of musical controllers called Beatbugs. Beatbug players can record live musical input from MIDI and acoustic instruments and respond by transforming the recorded material in real-time, creating motif-and-variation call-and-response routines on the fly. A central computer system analyzes MIDI and audio played by expert players and allows novice Beatbug players to personalize the analyzed material using a variety of transformation algorithms. This paper presents the motivation for developing the iltur system, followed by a brief survey of pervious and related work that guided the definition of the project’s goals. We then present the hardware and software approaches that were taken to address these goals, as well as a couple of compositions that were written for the system. The paper ends with a discussion based on observations of players using the iltur system and a number of suggestions for future work.

OROBORO is a novel collaborative controller which focuses on musical performance as social experience by exploring synchronized actions of two musicians operating a single instrument. Each performer uses two paddle mechanisms – one for hand orientation sensing and one for servo-motor actuated feedback. We introduce a haptic mirror in which the movement of one performer’s sensed hand is used to induce movement of the partner’s actuated hand and vice versa. We describe theoretical motivation, and hardware/software implementation.

Improvisational play methods encourage collaboration, communication and connection amongst groups. Learning can also be enhanced by the basic elements of improvisation; spontaneity, flexibility, and a willingness to experiment and take risks. Soliciting input from a diverse range of perspectives and building an environment based on trust also stimulates the efficacy and creativity of the group process. Consequently, improvisational techniques provide valuable insight for teams prototyping with new media and are particularly applicable to innovation in the field of entertainment. We consider this paper a prelude to further discussion about improvisation and the ways it can be used to explore a variety of educational and entertainment applications from a musical and theatrical perspective.

In this paper, design scenarios made possible by the use of an interactive illuminated floor as the basis of an audiovisual environment are presented. By interfacing a network of pressure sensitive, light-emitting tiles with a 5.1 channel speaker system and requisite audio software, many avenues for collaborative expression emerge, as do heretofore unexplored modes of multiplayer music and dance gaming. By giving users light and sound cues that both guide and respond to their movement, a rich environment is created that playfully integrates the auditory, the visual, and the kinesthetic into a unified interactive experience. undertaken to push the limits of the confluence of interactive dance, music and gaming. Descriptions of these and future investigations are detailed below. QuickTime ™ and a TIFF (Uncompressed) decompressor are needed to see this picture.