In this paper we describe our efforts towards the development of live performance computer-based musical instrumentation. Our design criteria include initial ease of use coupled with a long term potential for virtuosity, minimal and low variance latency, and clear and simple strategies for programming the relationship between gesture and musical result. We present custom controllers and unique adaptations of standard gestural interfaces, a programmable connectivity processor, a communications protocol called Open Sound Control (OSC), and a variety of metaphors for musical control. We further describe applications of our technology to a variety of real musical performances and directions for future research.

With current state-of-the-art human movement tracking techology it is possible to represent in real-time most of the degrees of freedom of a (part of the) human body. This allows for the design of a virtual musical instrument (VMI), analogous to a physical musical instrument, as a gestural interface, that will however provide for much greater freedom in the mapping of movement to sound. A musical performer may access therefore the currently unexplored real-time capabilities of sound synthesis systems. In order to decrease the learning and adaptation needed and avoid injuries, the design must address the musculo-skeletal, neuro-motor and symbolic levels that are involved in the programming and control of human movement. The use of virtual musical instruments will likely result in new ways of making music and new musical styles. Introduction In figure 1 an attempt has been made to show the development of musical instruments. Acoustic instruments transduce movements of a performer into so...

This paper provides a review of gestural control of sound synthesis in the context of the design and evaluation of digital musical instruments. It discusses research in various areas related to this field and equally focuses on four main topics: analysis of music performers’ gestures, gestural capture technologies, real-time sound synthesis methods, and strategies for mapping gesture variables to sound synthesis input parameters. Finally, this approach is illustrated by presenting an application of this research to the control of digital audio effects.

Digital musical instruments do not depend on physical constraints faced by their acoustic counterparts, such as characteristics of tubes, membranes, strings, etc. This fact permits a huge diversity of possibilities regarding sound production, but on the other hand strategies to design and perform these new instruments need to be devised in order to provide the same level of control subtlety available in acoustic instruments. In this paper I review various topics related to gestural control of music using digital musical instruments and identify possible trends in this domain.

This research report describes an approach to parameter estimation for physical models of sound-generating systems using distal teachers and forward models, [Jordan and Rumelhart, 1992, Jordan, 1990]. The general problem is to find an inverse model of a sound-generating system that transforms sounds to action parameters; these parameters constitute a model-based description of the sound. We first show that a two-layer feed-forward model is capable of performing inverse mappings for a simple physical model of a string. We refer to this learning strategy as direct inverse modeling; it requires an explicit teacher and it is only suitable for convex regions of the parameter space. A model of two strings was implemented that had non-convex regions in its parameter space. We show how the direct modeling strategy failed at the task of learning the inverse model in this case and that forward models can be used, in conjunction with distal teachers, to bias the learning of an inverse model so th...

Abstract. This paper describes the design and implementation of a framework designed to aid collaborative development of a digital musical instrument mapping layer 1. The goal was to create a system that allows mapping between controller and sound parameters without requiring a high level of technical knowledge, and which needs minimal manual intervention for tasks such as configuring the network and assigning identifiers to devices. Ease of implementation was also considered, to encourage future developers of devices to adopt a compatible protocol. System development included the design of a decentralized network for the management of peer-to-peer data connections using OpenSound Control. Example implementations were constructed using several different programming languages and environments. A graphical user interface for dynamically creating, modifying, and destroying mappings between control data streams and synthesis parameters is also presented.

This paper presents the Library of Maps 1 toolbox to aid in the mapping of control parameters to sound synthesis parameters via strategies that result from a geometric representation of control. A set of objects have been created for Max/MSP and Jitter that allow the user to map arbitrary high-dimensional data from control to sound parameter space, and to visualize this through the use of Jitter and OpenGL. The mapping implementations are discussed and related to existing work.

mdhoffma at cs.princeton.edu A crucial set of decisions in digital musical instrument design deals with choosing mappings between parameters controlled by the performer and the synthesis algorithms that actually generate sound. Feature-based synthesis offers a way to parameterize audio synthesis in terms of the quantifiable perceptual characteristics, or features, the performer wishes the sound to take on. Techniques for accomplishing such mappings and enabling feature-based synthesis to be performed in real time are discussed. An example is given of how a real-time performance system might be designed to take advantage of feature-based synthesis’s ability to provide perceptually meaningful control over a large number of synthesis parameters.

This article presents ISEE, an intuitive sound editing environment, as a general sound synthesis model based on expert auditory perception and cognition of musical instruments. It discusses the backgrounds of current synthesizer user interface design and related timbre space research. Of the three principal parameters of sound---pitch, loudness and timbre---ISEE focuses on control of timbre, and affects only the range of pitch and loudness. Timbre is manipulated using four abstract timbre parameters: overtones, brightness, articulation and envelope. These abstract timbre parameters are implemented in different ways for different instruments. They define instrument spaces of which a hierarchy can be built to structure refinement of timbre parameter behavior. An Apple Macintosh implementation of ISEE is described. ISEE has four main advantages over traditional sound synthesis editors. Firstly, it allows musicians to control sound synthesis as they control their musical instrument: by continuous movement, reducing cognitive control load. Secondly, it uses timbre parameters identified by human experience instead of indirect and intricate synthesis model parameters. Thirdly, it integrates a librarian system in the sound synthesis model. Finally, it enables transparent use of several synthesis models at a time. Introduction

With ”Gesturefication ” we designate the creation of a mapping between gestural control and sound synthesis parameters based on the recording and analysis of a sound and related gestures. These gestures may have actually produced the sound or simply describe the sound from a particular point of view such as instrument performance, music