Traditional uses of virtual audio environments tend to focus on perceptually accurate acoustic representations. Though spatialization of sound sources is important, it is necessary to leverage control of the sonic representation when considering musical applications. The proposed framework allows for the creation of perceptually immersive scenes that function as musical instruments. Loudspeakers and microphones are modeled within the scene along with the listener/performer, creating a navigable 3D sonic space where sound sources and sinks process audio accordingtouser-defined spatial mappings.

An experimental study comparing different user interfaces for a virtual drum is reported. Virtual here means that the drum is not a physical object. 16 subjects played the drum on five different interfaces and two metronome patterns trying to match their hits to the metronome clicks. Temporal accuracy of the playing was evaluated. The subjects also rated the interfaces subjectively. The results show that hitting the drum alternately from both sides with motion going through the drum plate was less accurate than the traditional one sided hitting. A physical stick was more accurate than a virtual computer graphic stick. Visual feedback of the drum slightly increased accuracy compared to receiving only auditory feedback. Most subjects evaluated the physical stick to offer a better feeling and to be more pleasant than the virtual stick.

In an effort to find a better suited interface for musical performance, a novel approach has been discovered and developed. At the heart of this approach is the concept of physical interaction with sound in space, where sound processing occurs at various 3-D locations and sending sound signals from one area to another is based on physical models of sound propagation. The control is based on a gestural vocabulary that is familiar to users, involving natural spatial interaction such as translating, rotating, and pointing in 3-D. This research presents a framework to deal with realtime control of 3-D audio, and describes how to construct audio scenes to accomplish various musical tasks. The generality and effectiveness of this approach has enabled us to reimplement several conventional applications, with the benefit of a substantially more powerful interface, and has further led to the conceptualization of several novel applications. 1.

Abstract. This paper presents a vision-based method for tracking guitar fingerings played by guitar players from stereo cameras. We propose a novel framework for colored finger markers tracking by integrating a Bayesian classifier into particle filters, with the advantages of performing automatic track initialization and recovering from tracking failures in a dynamic background. ARTag (Augmented Reality Tag) is utilized to calculate the projection matrix as an online process which allow guitar to be moved while playing. By using online adaptation of color probabilities, it is also able to cope with illumination changes.

3D graphical interaction offers a large amount of possibilities for musical applications. However it also carries several limitations that prevent it from being used as an efficient musical instrument. For example, input devices for 3D interaction or new gaming devices are usually based on 3 or 6 degrees-of-freedom tracking combined with push-buttons or joysticks. While buttons and joysticks do not provide good resolution for musical gestures, graphical interaction using tracking may bring enough expressivity but is weakened by accuracy and haptic feedback problems. Moreover, interaction based solely on tracking limit the possibilities brought by graphical interfaces in terms of musical gestures. We propose a new approach that separates the input modalities according to traditional musical gestures. This allows to combine the possibilities of graphical interaction as selection and modulation gestures with the accuracy and the expressivity of musical interaction as excitation gestures. We implement this approach with a new input device, Piivert, which combines 6DOF tracking and pressure detection. We describe associated interaction techniques and show how this new device can be valuable for immersive musical applications. 1

live-looping

Abstract. While virtual reality and 3D interaction provide new possibilities for musical applications, the existing immersive virtual instruments are limited to single process instruments or musical navigation tools. In this paper we present the 3D reactive widgets. These graphical elements enable simultaneous control and visualization of musical processes in 3D immersive environments. They also rely on the live-looping technique, allowing to build complex musical sequences. We describe the interaction techniques that we have designed and implemented to manipulate these widgets, including a virtual ray and tunnels. After having expressed the lack of expressivity and efficiency of the existing input devices for sound production gestures, we finally set the requirements for an appropriate device for musical interaction in 3D immersive environments. 1.