We present the architecture, technology and experimental applications of a real-time, multi-site, interactive and collaborative environment called Distributed Immersive Performance (DIP). The objective of DIP is to develop the technology for live, interactive musical performances in which the participants - subsets of musicians, the conductor and the audience - are in different physical locations and are interconnected by very high fidelity multichannel audio and video links. DIP is a specific realization of broader immersive technology - the creation of the complete aural and visual ambience that places a person or a group of people in a virtual space where they can experience events occurring at a remote site or communicate naturally regardless of their location. The DIP experimental system has interaction sites and servers in different locations on the USC campus and at several partners, including the New World Symphony of Miami Beach, FL. The sites have different types of equipment to test the effects of video and audio fidelity on the ease of use and functionality for different applications. Many sites have high-definition (HD) video or digital video (DV) quality images projected onto wide screen wall displays completely integrated with an immersive audio reproduction system for a seamless, fully three-dimensional aural environment with the correct spatial sound localization for participants. The system is capable of storage and playback of the many streams of synchronized audio and video data (immersidata), and utilizes novel protocols for the low-latency, seamless, synchronized realtime delivery of immersidata over local area networks and widearea networks such as Internet2. We discuss several recent interactive experiments using the system and many technical cha...

We present a spatialized audio rendering system for the use in immersive virtual environments. The system is optimized for renderhag a sufficient number of dynamically moving sound sources in multi-speaker environments using off-the-shelf audio hardware. Based on simplified physics-based models, we achieve a good trade-off between audio quality, spatial precision, and performance. Convincing acoustic room simulation is accomplished by integrating standard hardware reverberation devices as used in the professional audio and broadcast community. We elaborate on important design principles for audio rendering as well as on practical implementation issues. Moreover, we describe the integration of the audio rendering pipeline into a scene graph-based virtual reality toolkit.

In order to enable the user of a virtual reality system to be fully immersed in the virtual environment, the user must be presented with believable sensory input. Although the majority of virtual environments place the emphasis on visual cues, replicating the complex interactions of sound within an environment will benefit the level of immersion and hence the user's sense of presence. Three dimensional (spatial) sound systems allow a listener to perceive the position of sound sources, and the e#ect of the interaction of sound sources with the acoustic structure of the environment. This paper reviews the relevant biological and technical literature relevant to the generation of accurate acoustic displays for virtual environments, beginning with an introduction to the process of auditory perception in humans. This paper then critically examines common methods and techniques that have been used in the past as well as methods and techniques which are currently being used to generate spatial sound. In the process of doing so, the limitations, drawbacks, advantages and disadvantages associated with these techniques are also presented.

In this paper we describe the underlying concepts behind the spatial sound renderer built at the University of Southern California’s Immersive Audio Laboratory. In creating this sound rendering system, we were faced with three main challenges. First the rendering of sound using the Head-Related Transfer Functions, second the cancellation of the crosstalk terms and third the localization of the listener’s ears. To deal with the spatial rendering sound we use a two-layer method of modeling the HRTF’s. The first layer accurately reproduces the ITD’s and IAD’s, and the second layer reproduces the spectral characteristics of the HRTF’s. A novel method for generating the required crosstalk cancellation filters as the listener moves was developed based on Low-Rank modeling. Using Karhunen-Loeve expansion we can interpolate among listener positions from a small number of HRTF measurements. Finally we present a Head Detection algorithm for tracking the location of the listener’s ears in real time using a laser scanner. 1

Abstract—For many years, spatial (3D) sound using headphones has been widely used in a number of applications. A rich spatial sensation is obtained by using head related transfer functions (HRTF) and playing the appropriate sound through headphones. In theory, loudspeaker audio systems would be capable of rendering 3D sound fields almost as rich as headphones, as long as the room impulse responses (RIRs) between the loudspeakers and the ears are known. In practice, however, obtaining these RIRs is hard, and the performance of loudspeaker based systems is far from perfect. New hope has been recently raised by a system that tracks the user’s head position and orientation, and incorporates them into the RIRs estimates in real time. That system made two simplifying assumptions: it used generic HRTFs, and it ignored room reverberation. In this paper we tackle the second problem: we incorporate a room reverberation estimate into the RIRs. Note that this is a nontrivial task: RIRs vary significantly with the listener’s positions, and even if one could measure them at a few points, they are notoriously hard to interpolate. Instead, we take an indirect approach: we model the room, and from that model we obtain an estimate of the main reflections. Position and characteristics of walls do not vary with the users ’ movement, yet they allow to quickly compute an estimate of the RIR for each new user position. Of course the key question is whether the estimates are good enough. We show an improvement in localization perception of up to 32 % (i.e., reducing average error from 23.5 ◦ to 15.9 ◦).

Abstract—Traditional 3D audio systems using two loudspeakers often have a limited sweet spot and may suffer from poor performance in reverberant environments. This paper presents a novel binaural 3D audio system that actively combines head tracking and room modeling into 3D audio synthesis. The user’s head position and orientation are first tracked by a webcambased 3D head tracker. The system then improves its robustness to head movement and strong early reflections by incorporating the tracking information and an explicit room model into the binaural synthesis and crosstalk cancellation process. Sensitivity analysis on the room model shows that the method is reasonably robust to modeling errors. Subjective listening tests confirm that the proposed 3D audio system significantly improves the users’ perception and ability for localization. Index Terms—3D audio, head tracking, room modeling, loudspeaker, spatial audio. I.

this report is to present an overview of some of the methods and techniques employed by 3D (spatial) sound systems in order to position virtual sound sources arbitrarily in three dimensional space. In addition, the problems and limitations associated with these techniques as well as potential solutions to these drawbacks will also be described. However, prior to discussing 3D sound technologies, given the importance of an understanding of the perception of sound and sound localization, this report begins with a brief introduction on the physical attributes of sound and how these attributes can be measured, followed by an elaborate discussion on the primary human auditory localization cues. Several auditory phenomena such as the precedence e#ect and auditory distance perception are introduced and will also be described. Chapter 2 will present a brief history of the methods used to convey sound to listeners using loudspeakers, beginning with an introduction of monaural based systems, two channel stereo based systems, Quadraphonics, Ambisonics and surround sound system. Chapter 3 will focus on the simulation of human auditory localization in a virtual environment. In particular, this chapter will describe techniques and models in order to re-create the cues available in our "natural" listening environment, including models to predict the ITD. methods of measuring the HRTF for a specific position and techniques to model reverberation. Several techniques for conveying sound in a virtual environment using loudspeakers and the problems and drawbacks associated with these techniques is presented in Chapter 4, including transaural audio and amplitude panning. In addition, this chapter will also discuss some of the issues related to the presentation of spatial audio using headpho...

In this paper, we propose a reconfigurable and scalable hardware accelerator for 3D-audio systems based on the Wave Field Synthesis technology. Previous related work reveals that WFS sound systems are based on using standard PCs. However, two major obstacles are the relative low number of real-time sound sources that can be processed and the high power consumption. The proposed accelerator alleviates these limitations by its performance and energy efficient design. We propose a scalable organization comprising multiple rendering units (RUs), each of them independently processing audio samples. The processing is done in an environment of continuously varying number of sources and speakers. We provide a comprehensive study on the design trade-offs with respect to this multiplicity of sources and speakers. A hardware prototype of our proposal was implemented on a Virtex4FX60 FPGA operating at 200 MHz. A single RU can achieve up to 7x WFS processing speedup compared to a software implementation running on a Pentium D at 3.4 GHz, while consuming, according to Xilinx XPower, approximately 3 W of power only. 1.

To be immersed in a virtual environment, the user must be presented with plausible sensory input including auditory cues. A virtual (three-dimensional) audio display aims to allow the user to perceive the position of a sound source at an arbitrary position in three-dimensional space despite the fact that the generated sound may be emanating from a fixed number of loudspeakers at fixed positions in space or a pair of headphones. The foundation of virtual audio rests on the development of technology to present auditory signals to the listener’s ears so that these signals are perceptually equivalent to those the listener would receive in the environment being simulated. This paper reviews the human perceptual and technical literature relevant to the modeling and generation of accurate audio displays for virtual environments. Approaches to acoustical environment simulation are summarized and the A virtual (three-dimensional) audio display allows a listener to perceive the position of a sound source, emanating from a fixed number of stationary loudspeakers or a pair of headphones, as coming from an arbitrary location in three-dimensional space. Spatial sound technology goes far beyond traditional stereo and surround sound techniques by