We present fast methods for separating the direct and global illumination components of a scene measured by a camera and illuminated by a light source. In theory, the separation can be done with just two images taken with a high frequency binary illumination pattern and its complement. In practice, a larger number of images are used to overcome the optical and resolution limitations of the camera and the source. The approach does not require the material properties of objects and media in the scene to be known. However, we require that the illumination frequency is high enough to adequately sample the global components received by scene points. We present separation results for scenes that include complex interreflections, subsurface scattering and volumetric scattering. Several variants of the separation approach are also described. When a sinusoidal illumination pattern is used with different phase shifts, the separation can be done using just three images. When the computed images are of lower resolution than the source and the camera, smoothness constraints are used to perform the separation using a single image. Finally, in the case of a static scene that is lit by a simple point source, such as the sun, a moving occluder and a video camera can be used to do the separation. We also show several simple examples of how novel images of a scene can be computed from the separation results.

Figure 1: (a) Conventional photograph of a scene, illuminated by a projector with all its pixels turned on. (b) After measuring the light transport between the projector and the camera using structured illumination, our technique is able to synthesize a photorealistic image from the point of view of the projector. This image has the resolution of the projector and is illuminated by a light source at the position of the camera. The technique can capture subtle illumination effects such as caustics and self-shadowing. Note, for example, how the glass bottle in the primal image (a) appears as the caustic in the dual image (b) and vice-versa. Because we have determined the complete light transport between the projector and camera, it is easy to relight the dual image using a synthetic light source (c) or a light modified by a matte captured later by the same camera (d). We present a novel photographic technique called dual photography, which exploits Helmholtz reciprocity to interchange the lights and cameras in a scene. With a video projector providing structured illumination, reciprocity permits us to generate pictures from the viewpoint of the projector, even though no camera was present at that location. The technique is completely image-based, requiring no knowledge of scene geometry or surface properties, and

We propose a novel method for estimating the surface shapes of transparent objects by analyzing the polarization state of the light. Existing methods do not fully consider the reflection, refraction, and transmission of the light occurring inside a transparent object. We employ a polarization raytracing method to compute both the path of the light and its polarization state. Our proposed iterative computation method estimates the surface shape of the transparent object by minimizing the difference between the polarization data rendered by the polarization raytracing method and the polarization data obtained from a real object.

The scanning of 3D geometry has become a popular way of capturing the shape of real-world objects. Transparent objects, however, pose problems for traditional scanning methods. We present a tomographic method for recovering the shape of objects made of

We present a technique for interactive rendering of glossy objects in complex and dynamic lighting environments that captures interreflections and all-frequency shadows. Our system is based on precomputed radiance transfer and separable BRDF approximation. We factor glossy BRDFs using a separable decomposition and keep only a few low-order approximation terms, each consisting of a purely view-dependent and a purely light-dependent component. In the precomputation step, for every vertex we sample its visibility and compute a direct illumination transport vector corresponding to each BRDF term. We use modern graphics hardware to accelerate this step, and further compress the data using a non-linear wavelet approximation. The direct illumination pass is followed by one or more interreflection passes, each of which gathers compressed transport vectors from the previous pass to produce global illumination transport vectors. To render at run time, we dynamically sample the lighting to produce a light vector, also represented in a wavelet basis. We compute the inner product of the light vector with the precomputed transport vectors, and the results are further combined with the BRDF view-dependent components to produce vertex colors. We describe acceleration of the rendering algorithm using programmable graphics hardware, and discuss the limitations and tradeoffs imposed by the hardware.

We present a novel acquisition device to capture high resolution 4D reflectance fields of real scenes. The device consists of a concave hemispherical surface coated with a rough specular paint and a digital video projector with a fish-eye lens positioned near the center of the hemisphere. The scene is placed near the projector, also near the center, and photographed from a fixed vantage point. The projector projects a high-resolution image of incident illumination which is reflected by the rough hemispherical surface to become the illumination on the scene. We demonstrate the utility of this device by capturing a high resolution hemispherical reflectance field of a specular object which would be difficult to capture using previous acquisition techniques. 1 1

This state of the art report covers reconstruction methods for transparent and specular objects or phenomena. While the 3D acquisition of opaque surfaces with lambertian reflectance is a well-studied problem, transparent, refractive, specular and potentially dynamic scenes pose challenging problems for acquisition systems. This report reviews and categorizes the literature in this field. Despite tremendous interest in object digitization, the acquisition of digital models of transparent or specular objects is far from being a solved problem. On the other hand, real-world data is in high demand for applications such as object modeling, preservation of historic artifacts and as input to data driven modeling techniques. With this report we aim at providing a reference for and an introduction to the field of transparent and specular object reconstruction. We describe acquisition approaches for different classes of objects. Transparent objects/phenomena that do not change the straight ray geometry can be found foremost in natural phenomena. Refraction effects are usually small and can be considered negligible for these objects. Phenomena as diverse as fire, smoke, and interstellar nebulae can be modeled using a straight ray model of image formation. Refractive and specular surfaces on the other hand change the straight rays into usually piecewise linear ray paths, adding additional complexity to the reconstruction problem. Translucent objects exhibit significant sub-surface scattering effects rendering traditional acquisition approaches unstable. Different classes of techniques have been developed to deal with these problems and good reconstruction results can be achieved with current state-of-the-art techniques. However, the approaches are still specialized and targeted at very specific object classes. We classify the existing literature and hope to provide an entry point to this exiting field.

The accurate measurement of the light transport characteristics of a complex scene is an important goal in computer graphics and has applications in relighting and dual photography. However, since the light transport data sets are typically very large, much of the previous research has focused on adaptive algorithms that capture them efficiently. In this work, we propose a novel, non-adaptive algorithm that takes advantage of the compressibility of the light transport signal in a transform domain to capture it with less acquisitions than with standard approaches. To do this, we leverage recent work in the area of compressed sensing, where a signal is reconstructed from a few samples assuming that it is sparse in a transform domain. We demonstrate our approach by performing dual photography and relighting by using a much smaller number of acquisitions than would normally be needed. Because our algorithm is not adaptive, it is also simpler to implement than many of the current approaches.

With image-based relighting (IBL), one can render realistic relit images of a scene without prior knowledge of object geometry in the scene. However, traditional IBL methods require a large number of basis images, each corresponding to a lighting pattern, to estimate the surface reflectance function (SRF) of the scene. In this paper, we present a statistical approach to estimating the SRF which requires fewer basis images. We formulate the SRF estimation problem in a signal reconstruction framework. We use the principal component analysis (PCA, [1]) to show that the most effective lighting patterns for the data acquisition process are the eigenvectors of the covariance matrix of the SRFs, corresponding to the largest eigenvalues. In addition, we show that for typical SRFs, especially when the objects have Lambertian surfaces, DCT-based lighting patterns perform as well as the optimal PCA-based lighting patterns. We compare SRF estimation performance of the statistical approach with traditional IBL techniques. Experimental results show that the statistical approach can achieve better performance with fewer basis images. 1.

New Models and Methods for Matting and Compositing by Yung-Yu Chuang Co-Chairs of Supervisory Committee: Professor Brian Curless Computer Science & Engineering Professor David H. Salesin Computer Science & Engineering Matting and compositing are fundamental operations in graphics and visual effects. Despite having enjoyed wide usage for many years, traditional matting and compositing have limitations. Traditional matting methods either require special setups or cannot handle objects with complex silhouettes. Furthermore, the traditional compositing model is effective in modeling color blending effects but not reflection, refraction, and shadows. In this dissertation, we address these limitations and present a set of new compositing models and matting methods. To pull mattes of complex silhouettes from natural images, we introduce a principled statistical approach called Bayesian image matting. We also extend this algorithm to handle video sequences with the help of optical flow computation and background estimation. On the compositing side, previous work on environment matting has been shown to handle refraction and reflection, but the resulting mattes are not very accurate. We propose a more accurate environment matting model and method that requires using more images. For shadows, we develop a physically-motivated shadow compositing equation. Based on this equation, we introduce a shadow matting method for extracting shadow mattes from videos with natural backgrounds, and we demonstrate a novel process for acquiring the photometric and geometric properties of the background to enable creation of realistic shadow composites. Finally, we present a novel application of Bayesian image matting for animating still pictures.