This paper describes the first system designed to allow children to conduct an audio and video recording of an orchestra. No prior music experience is required to control the orchestra, and the system uses an advanced algorithm to time stretch the audio in real-time at high quality and without altering the pitch. We will discuss the requirements and challenges of designing an interface to target our particular user group (children), followed by some system implementation details. An overview of the algorithm used for audio time stretching will also be presented. We are currently using this technology to study and compare professional and non-professional conducting behavior, and its implications when designing new interfaces for multimedia. You're the Conductor is currently a successful exhibit at the Children's Museum in Boston, USA.

Inertial measurement components, which sense either acceleration or angular rate, are being embedded into common user interface devices more frequently as their cost continues to drop dramatically. These devices hold a number of advantages over other sensing technologies: they measure relevant parameters for human interfaces and can easily be embedded into wireless, mobile platforms. The work in this dissertation demonstrates that inertial measurement can be used to acquire rich data about human gestures, that we can derive efficient algorithms for using this data in gesture recognition, and that the concept of a parameterized atomic gesture recognition has merit. Further we show that a framework combining these three levels of description can be easily used by designers to create robust applications.

Abstract. This paper presents a virtual embodied agent that can conduct musicians in a live performance. The virtual conductor conducts music specified by a MIDI file and uses input from a microphone to react to the tempo of the musicians. The current implementation of the virtual conductor can interact with musicians, leading and following them while they are playing music. Different time signatures and dynamic markings in music are supported. 1

This paper describes the first multimedia system that allows users to conduct a realistic electronic orchestra. Users control tempo, dynamics, and instrument emphasis of the orchestra through natural conducting gestures with an infrared baton. Using gesture recognition and tempo adjustment algorithms, the system plays back an audio and video recording of an actual orchestra that follows the user's conducting in real time.

Designing a conducting gesture analysis system for public spaces poses unique challenges. We present conga, a software framework that enables automatic recognition and interpretation of conducting gestures. conga is able to recognize multiple types of gestures with varying levels of difficulty for the user to perform, from a standard four-beat pattern, to simplified up-down conducting movements, to no pattern at all. conga provides an extendable library of feature detectors linked together into a directed acyclic graph; these graphs represent the various conducting patterns as gesture profiles. At run-time, conga searches for the best profile to match a user’s gestures in real-time, and uses a beat prediction algorithm to provide results at the sub-beat level, in addition to output values such as tempo, gesture size, and the gesture’s geometric center. Unlike some previous approaches, conga does not need to be trained with sample data before use. Our preliminary user tests show that conga has a beat recognition rate of over 90%. conga is deployed as the gesture recognition system for Maestro!, an interactive conducting exhibit that opened in the

Abstract. This paper describes affective and psychophysiological foundations used to help to control affective content in music production. Our work includes the proposal of a knowledge base grounded on the state of the art done in areas of Music Psychology. This knowledge base has relations between affective states (happiness, sadness, etc.) and high level music features (rhythm, melody, etc.) to assist in the production of affective music. A computer system uses this knowledge base to select and transform chunks of music. The methodology underlying this system is essentially founded on Affective Computing topics. Psychophysiology measures will be used to detect listener’s affective state. 1

An important feature of the behaviour of human agents is their use of emotional factors in communication and determination of be2 haviour. The incorporation of emotional or aective features is an important topic in the development of Intelligent Virtual Agents (IVAs) for operation in Virtual Environments (VEs). A number of precepts can be identied which should be considered by a designer wishing to include aective states in IVAs. These precepts include: considering how emotion can actually be communicated rather than just using emotion as an incidental \add-on" which merely increases the presence of an environment; realizing that aective information can be utilised in VEs which are comparatively simple; that aective communication in VEs is not always about directly reproducing the emotional communications used in human face-to-face communication; that emotion recognition can be placed at various locations in the system; that incorporating aect models might lead to t...

This paper describes the artistic projects undertaken at Immersion Music, Inc. (www.immersionmusic.org) during its three-year existence. We detail work in interactive performance systems, computer-based training systems, and concert production.

The galvanic skin response (GSR), also known as electrodermal response, measures changes in electrical resistance across two regions of the skin. Galvanic skin response can measure arousal levels in children with autism; however, the GSR signal may be overwhelmed by the vigorous movements of the children. This paper introduces ActionGSR, a wireless sensor capable of measuring both GSR and acceleration simultaneously in an attempt to disambiguate valid GSR signals from motion artifacts. 1

Discussion of time in interactive computer music systems engineering has been largely limited to data acquisition rates and latency. Since music is an inherently time-based medium, we believe that time plays a more important role in both the usability and implementation of these systems. In this paper, we present a time design space, which we use to expose some of the challenges of developing computer music systems with time-based interaction. We describe and analyze the time-related issues we encountered whilst designing and building a series of interactive music exhibits that fall into this design space. These issues often occur because of the varying and sometimes conflicting conceptual models of time in the three domains of user, application (music), and engineering. We present some of our latest work in conducting gesture interpretation and frameworks for digital audio, which attempt to analyze and address these conflicts in temporal conceptual models.