Abstract. Image-based relighting (IBL) is a technique to change the illumination of an image-based object/scene. In this paper, we define a representation called the reflected irradiance field which records the light reflected from a scene as viewed at a fixed viewpoint as a result of moving a point light source on a plane. It synthesizes a novel image under a different illumination by interpolating and superimposing appropriate recorded samples. Furthermore, we study the minimum sampling problem of the reflected irradiance field, i.e., how many light source positions are needed. We find that there exists a geometry-independent bound for the sampling interval whenever the second-order derivatives of the surface BRDF and the minimum depth of the scene are bounded. This bound ensures that when the novel light source is on the plane, the error in the reconstructed image is controlled by a given tolerance, regardless of the geometry. We also analyze the bound of depth error so that the extra reconstruction error can also be governed when the novel light source is off-plane. Experiments on both synthetic and real surfaces are conducted to verify our analysis. Keywords: sampling, BRDF, light field, Lumigraph, plenoptic functions, image-based rendering, relighting

A framework for analyzing distortions in non-single viewpoint imaging systems is presented. Such systems possess loci of viewpoints called caustics. In general, perspective (or undistorted) views cannot be computed from images acquired with such systems without knowing scene structure. Views computed without scene structure will exhibit distortions which we call caustic distortions. We first introduce a taxonomy of distortions based on the geometry of imaging systems. Then, we derive a metric to quantify caustic distortions. We present an algorithm to compute minimally distorted views using simple priors on scene structure. These priors are defined as parameterized primitives such as spheres, planes and cylinders with simple uncertainty models for the parameters. To

To store and transmit the large amount of image data necessary for Image-based Rendering (IBR), efficient coding schemes are required. This paper presents two different approaches which exploit 3-D scene geometry for multi-view compression. In texture-based coding, images are converted to view-dependent texture maps for compression. In model- aided predictive coding, scene geometry is used for disparity compensation and occlusion detection between images. While both coding strategies are able to attain compression ratios exceeding 2000:1, individual coding performance is found to depend on the accuracy of the available geometry model. Experiments with real-world as well as synthetic image sets show that texture-based coding is more sensitive to geometry inaccuracies than predictive coding. A rate- distortion theoretical analysis of both schemes supports these findings. For reconstructed approximate geometry models, model-aided predictive coding performs best, while texture-based coding yields superior coding results if scene geometry is exactly known.

We propose a novel approach for light field compression that incorporates disparity compensation into 4-D wavelet coding using disparity-compensated lifting. With this approach, we obtain the benefits of wavelet coding, including compression efficiency and scalability in all dimensions. Additionally, our proposed approach solves the irreversibility limitations of previous wavelet coding approaches. Experimental results show that the compression efficiency of the proposed technique outperforms current state-of-theart wavelet coding techniques by a wide margin.

We consider the problem of calibrating a highly generic imaging model, that consists of a non-parametric association of a projection ray in 3D to every pixel in an image. Previous calibration approaches for this model do not seem to be directly applicable for cameras with large fields of view and non-central cameras. In this paper, we describe a complete calibration approach that should in principle be able to handle any camera that can be described by the generic imaging model. Initial calibration is performed using multiple images of overlapping calibration grids simultaneously. This is then improved using pose estimation and bundle adjustment-type algorithms. The approach has been applied on a wide variety of central and non-central cameras including fisheye lens, catadioptric cameras with spherical and hyperbolic mirrors, and multi-camera setups. We also consider the question if non-central models are more appropriate for certain cameras than central models. 1.

Abstract—A robust synthesis method is proposed to automatically infer missing color and texture information from a damaged 2D image by ND tensor voting (N>3). The same approach is generalized to range and 3D data in the presence of occlusion, missing data and noise. Our method translates texture information into an adaptive ND tensor, followed by a voting process that infers noniteratively the optimal color values in the ND texture space. A two-step method is proposed. First, we perform segmentation based on insufficient geometry, color, and texture information in the input, and extrapolate partitioning boundaries by either 2D or 3D tensor voting to generate a complete segmentation for the input. Missing colors are synthesized using ND tensor voting in each segment. Different feature scales in the input are automatically adapted by our tensor scale analysis. Results on a variety of difficult inputs demonstrate the effectiveness of our tensor voting approach. Index Terms—Image restoration, segmentation, color, texture, tensor voting, applications. 1

Stereo correspondence algorithms typically produce a single depth map. In addition to the usual problems of occlusions and textureless regions, such algorithms cannot model the variation in scene or object appearance with respect to the viewing position. In this paper, we propose a new representation that overcomes the appearance variation problem associated with an image sequence. Rather than estimating a single depth map, we associate a depth map with each input image (or a subset of them). Our representation is motivated by applications such as view interpolation and depth-based segmentation for model-building or layer extraction. We describe two approaches to extract such a representation from a sequence of images. The first approach, which is more classical, computes the local depth map associated with each chosen reference frame independently. The novelty of this approach lies in its combination of shiftable windows, temporal selection, and graph cut optimization. The second approach simultaneously optimizes a set of self-consistent depth maps at multiple key-frames. Since multiple depth maps are estimated simultaneously, visibility can be modeled explicitly and disparity consistency imposed across the different depth maps. Results, which include a difficult specular scene example, show the effectiveness of our proposals.

A long-standing research problem in computer graphics is to reproduce the visual experience of walking through a large photorealistic environment interactively. On one hand, traditional geometry-based rendering systems fall short of simulating the visual realism of a complex environment. On the other hand, image-based rendering systems have to date been unable to capture and store a sampled representation of a large environment with complex lighting and visibility effects.

This paper proposes a novel technique to computing geometric information from images captured under parallel projections. Parallel images are desirable for stereo reconstruction because parallel projection significantly reduces foreshortening. As a result, correlation based matching becomes more effective. Since parallel projection cameras are not commonly available, we construct parallel images by rebinning a large sequence of perspective images. Epipolar geometry, depth recovery and projective invariant for both 1D and 2D parallel stereos are studied. From the uncertainty analysis of depth reconstruction, it is shown that parallel stereo is superior to both conventional perspective stereo and the recently developed multiperspective stereo for vision reconstruction, in that uniform reconstruction error is obtained in parallel stereo. Traditional stereo reconstruction techniques, e.g. multi-baseline stereo, can still be applicable to parallel stereo without any modifications because epipolar lines in a parallel stereo are perfectly straight. Experimental results further confirm the performance of our approach.

It is generally recognized that 3-D models are compact representations for rendering. While pure image-based rendering techniques are capable of producing highly photorealistic outputs, the size of the input "model" is usually very large. The important issues in trading off geometry versus images include compactness of representation, photorealism of reconstructed views, and speed of rendering. In this paper, we describe our past work in modeling and rendering, and articulate lessons learnt. We then delineate our vision of an ideal rendering system. 1 Introduction Many commercial graphics systems and research focus on explicit 3-D modeling. This is a very traditional area, and the 3-D modeling and rendering issues have been mostly dealt with successfully. It is generally true that 3-D models are compact representations for rendering, especially when fast specialized hardware graphics accelerators are readily available. Better photorealism can be achieved at a loss in rendering speed....