Ray geometry in non-pinhole cameras: a survey
---
paper_title: Multiple view geometry in computer vision

paper_content:

From the Publisher: ::: A basic problem in computer vision is to understand the structure of a real world scene given several images of it. Recent major developments in the theory and practice of scene reconstruction are described in detail in a unified framework. The book covers the geometric principles and how to represent objects algebraically so they can be computed and applied. The authors provide comprehensive background material and explain how to apply the methods and implement the algorithms directly.
---
paper_title: Fourier depth of field

paper_content:

Optical systems used in photography and cinema produce depth-of-field effects, that is, variations of focus with depth. These effects are simulated in image synthesis by integrating incoming radiance at each pixel over the lense aperture. Unfortunately, aperture integration is extremely costly for defocused areas where the incoming radiance has high variance, since many samples are then required for a noise-free Monte Carlo integration. On the other hand, using many aperture samples is wasteful in focused areas where the integrand varies little. Similarly, image sampling in defocused areas should be adapted to the very smooth appearance variations due to blurring. This article introduces an analysis of focusing and depth-of-field in the frequency domain, allowing a practical characterization of a light field's frequency content both for image and aperture sampling. Based on this analysis we propose an adaptive depth-of-field rendering algorithm which optimizes sampling in two important ways. First, image sampling is based on conservative bandwidth prediction and a splatting reconstruction technique ensures correct image reconstruction. Second, at each pixel the variance in the radiance over the aperture is estimated and used to govern sampling. This technique is easily integrated in any sampling-based renderer, and vastly improves performance.
---
paper_title: Light field rendering

paper_content:

A number of techniques have been proposed for flying through scenes by redisplaying previously rendered or digitized views. Techniques have also been proposed for interpolating between views by warping input images, using depth information or correspondences between multiple images. In this paper, we describe a simple and robust method for generating new views from arbitrary camera positions without depth information or feature matching, simply by combining and resampling the available images. The key to this technique lies in interpreting the input images as 2D slices of a 4D function the light field. This function completely characterizes the flow of light through unobstructed space in a static scene with fixed illumination. We describe a sampled representation for light fields that allows for both efficient creation and display of inward and outward looking views. We hav e created light fields from large arrays of both rendered and digitized images. The latter are acquired using a video camera mounted on a computer-controlled gantry. Once a light field has been created, new views may be constructed in real time by extracting slices in appropriate directions. Since the success of the method depends on having a high sample rate, we describe a compression system that is able to compress the light fields we have generated by more than a factor of 100:1 with very little loss of fidelity. We also address the issues of antialiasing during creation, and resampling during slice extraction. CR Categories: I.3.2 [Computer Graphics]: Picture/Image Generation — Digitizing and scanning, Viewing algorithms; I.4.2 [Computer Graphics]: Compression — Approximate methods Additional keywords: image-based rendering, light field, holographic stereogram, vector quantization, epipolar analysis
---
paper_title: Multiple view geometry in computer vision

paper_content:

From the Publisher: ::: A basic problem in computer vision is to understand the structure of a real world scene given several images of it. Recent major developments in the theory and practice of scene reconstruction are described in detail in a unified framework. The book covers the geometric principles and how to represent objects algebraically so they can be computed and applied. The authors provide comprehensive background material and explain how to apply the methods and implement the algorithms directly.
---
paper_title: Unstructured lumigraph rendering

paper_content:

We describe an image based rendering approach that generalizes many current image based rendering algorithms, including light field rendering and view-dependent texture mapping. In particular, it allows for lumigraph-style rendering from a set of input cameras in arbitrary configurations (i.e., not restricted to a plane or to any specific manifold). In the case of regular and planar input camera positions, our algorithm reduces to a typical lumigraph approach. When presented with fewer cameras and good approximate geometry, our algorithm behaves like view-dependent texture mapping. The algorithm achieves this flexibility because it is designed to meet a set of specific goals that we describe. We demonstrate this flexibility with a variety of examples.
---
paper_title: Light field rendering

paper_content:

A number of techniques have been proposed for flying through scenes by redisplaying previously rendered or digitized views. Techniques have also been proposed for interpolating between views by warping input images, using depth information or correspondences between multiple images. In this paper, we describe a simple and robust method for generating new views from arbitrary camera positions without depth information or feature matching, simply by combining and resampling the available images. The key to this technique lies in interpreting the input images as 2D slices of a 4D function the light field. This function completely characterizes the flow of light through unobstructed space in a static scene with fixed illumination. We describe a sampled representation for light fields that allows for both efficient creation and display of inward and outward looking views. We hav e created light fields from large arrays of both rendered and digitized images. The latter are acquired using a video camera mounted on a computer-controlled gantry. Once a light field has been created, new views may be constructed in real time by extracting slices in appropriate directions. Since the success of the method depends on having a high sample rate, we describe a compression system that is able to compress the light fields we have generated by more than a factor of 100:1 with very little loss of fidelity. We also address the issues of antialiasing during creation, and resampling during slice extraction. CR Categories: I.3.2 [Computer Graphics]: Picture/Image Generation — Digitizing and scanning, Viewing algorithms; I.4.2 [Computer Graphics]: Compression — Approximate methods Additional keywords: image-based rendering, light field, holographic stereogram, vector quantization, epipolar analysis
---
paper_title: Multiperspective projection and collineation

paper_content:

We present theories of multiperspective projection and collineation. Given an arbitrary multiperspective imaging system that captures smoothly varying set of rays, we show how to map the rays onto a 2D ray manifold embedded in a 4D linear vector space. The characteristics of this imaging system, such as its projection, collineation, and image distortions can be analyzed by studying the 2D tangent planes of this ray manifold. These tangent planes correspond to the recently proposed general linear camera (GLC) model. In this paper, we study the imaging process of the GLCs. We show the GLC imaging process can be broken down into two separate stages: the mapping of 3D geometry to rays and the sampling of those rays over an image plane. We derive a closed-form solution to projecting 3D points in a scene to rays in a GLC. A GLC image is created by sampling these rays over an image plane. We develop a notion of GLC collineation analogous to pinhole cameras. GLC collineation describes the transformation between the images of a single GLC due to changes in sampling and image plane selection. We show that general GLC collineations can be characterized by a quartic (4th order) rational function. GLC projection and collineation provides a basis for developing new computer vision algorithms suitable for analyzing a wider range of imaging systems than current methods, based on simple pinhole projection models, permit.
---
paper_title: A theory of multi-perspective defocusing

paper_content:

We present a novel theory for characterizing defocus blurs in multi-perspective cameras such as catadioptric mirrors. Our approach studies how multi-perspective ray geometry transforms under the thin lens. We first use the General Linear Cameras (GLCs) [21] to approximate the incident multi-perspective rays to the lens and then apply a Thin Lens Operator (TLO) to map an incident GLC to the exit GLC. To study defocus blurs caused by the GLC rays, we further introduce a new Ray Spread Function (RSF) model analogous the Point Spread Function (PSF). While PSF models defocus blurs caused by a 3D scene point, RSF models blurs spread by rays. We derive closed form RSFs for incident GLC rays, and we show that for catadioptric cameras with a circular aperture, the RSF can be effectively approximated as a single or mixtures of elliptic-shaped kernels. We apply our method for predicting defocus blurs on commonly used catadioptric cameras and for reducing de-focus blurs in catadioptric projections. Experiments on synthetic and real data demonstrate the accuracy and general applicability of our approach.
---
paper_title: Fourier slice photography

paper_content:

This paper contributes to the theory of photograph formation from light fields. The main result is a theorem that, in the Fourier domain, a photograph formed by a full lens aperture is a 2D slice in the 4D light field. Photographs focused at different depths correspond to slices at different trajectories in the 4D space. The paper demonstrates the utility of this theorem in two different ways. First, the theorem is used to analyze the performance of digital refocusing, where one computes photographs focused at different depths from a single light field. The analysis shows in closed form that the sharpness of refocused photographs increases linearly with directional resolution. Second, the theorem yields a Fourier-domain algorithm for digital refocusing, where we extract the appropriate 2D slice of the light field's Fourier transform, and perform an inverse 2D Fourier transform. This method is faster than previous approaches.
---
paper_title: Recovering specular surfaces using curved line images

paper_content:

We present a new shape-from-distortion framework for recovering specular (reflective/refractive) surfaces. While most existing approaches rely on accurate correspondences between 2D pixels and 3D points, we focus on analyzing the curved images of 3D lines which we call curved line images or CLIs. Our approach models CLIs of local reflections or refractions using the recently proposed general linear cameras (GLCs). We first characterize all possible CLIs in a GLC. We show that a 3D line will appear as a conic in any GLC. For a fixed GLC, the conic type is invariant to the position and orientation of the line and is determined by the GLC parameters. Furthermore, CLIs under single reflection/refraction can only be lines or hyperbolas. Based on our new theory, we develop efficient algorithms to use multiple CLIs to recover the GLC camera parameters. We then apply the curvature-GLC theory to derive the Gaussian and mean curvatures from the GLC intrinsics. This leads to a complete distortion-based reconstruction framework. Unlike conventional correspondence-based approaches that are sensitive to image distortions, our approach benefits from the CLI distortions. Finally, we demonstrate applying our framework for recovering curvature fields on both synthetic and real specular surfaces.
---
paper_title: Epsilon Stereo Pairs

paper_content:

Human stereo vision works by fusing a pair of perspective images with a purely horizontal parallax. Recent developments suggest that very few varieties of multiperspective stereo pairs exist. In this paper, we introduce a new stereo model, which we call epsilon stereo pairs, for fusing a broader class of multiperspective images. An epsilon stereo pair consists of two images with a slight vertical parallax. We show many multiperspective camera pairs that do not satisfy the stereo constraint can still form epsilon stereo pairs. We then introduce a new ray-space warping algorithm to minimize stereo inconsistencies in an epsilon pair using multiperspective collineations. This makes epsilon stereo model a promising tool for synthesizing close-to-stereo fusions from many non-stereo pairs.
---
paper_title: Linear pushbroom cameras

paper_content:

Modeling and analyzing pushbroom sensors commonly used in satellite imagery is difficult and computationally intensive due to the motion of an orbiting satellite with respect to the rotating Earth, and the nonlinearity of the mathematical model involving orbital dynamics. In this paper, a simplified model of a pushbroom sensor (the linear pushbroom model) is introduced. It has the advantage of computational simplicity while at the same time giving very accurate results compared with the full orbiting pushbroom model. Besides remote sensing, the linear pushbroom model is also useful in many other imaging applications. Simple noniterative methods are given for solving the major standard photogrammetric problems for the linear pushbroom model: computation of the model parameters from ground-control points; determination of relative model parameters from image correspondences between two images; and scene reconstruction given image correspondences and ground-control points. The linear pushbroom model leads to theoretical insights that are approximately valid for the full model as well. The epipolar geometry of linear pushbroom cameras is investigated and shown to be totally different from that of a perspective camera. Nevertheless, a matrix analogous to the fundamental matrix of perspective cameras is shown to exist for linear pushbroom sensors. From this it is shown that a scene is determined up to an affine transformation from two views with linear pushbroom cameras.
---
paper_title: Mosaicing New Views: The Crossed-Slits Projection

paper_content:

We introduce anew kind of mosaicing, where the position of the sampling strip varies as a function of the input camera location. The new images that are generated this way correspond to a new projection model defined by two slits, termed here the Crossed-Slits (X-Slits) projection. In this projection model, every 3D point is projected by a ray defined as the line that passes through that point and intersects the two slits. The intersection of the projection rays with the imaging surface defines the image. X-Slits mosaicing provides two benefits. First, the generated mosaics are closer to perspective images than traditional pushbroom mosaics. Second, by simple manipulations of the strip sampling function, we can change the location of one of the virtual slits, providing a virtual walkthrough of a X-Slits camera; all this can be done without recovering any 3D geometry and without calibration. A number of examples where we translate the virtual camera and change its orientation are given; the examples demonstrate realistic changes in parallax, reflections, and occlusions.
---
paper_title: Linear pushbroom cameras

paper_content:

Modeling and analyzing pushbroom sensors commonly used in satellite imagery is difficult and computationally intensive due to the motion of an orbiting satellite with respect to the rotating Earth, and the nonlinearity of the mathematical model involving orbital dynamics. In this paper, a simplified model of a pushbroom sensor (the linear pushbroom model) is introduced. It has the advantage of computational simplicity while at the same time giving very accurate results compared with the full orbiting pushbroom model. Besides remote sensing, the linear pushbroom model is also useful in many other imaging applications. Simple noniterative methods are given for solving the major standard photogrammetric problems for the linear pushbroom model: computation of the model parameters from ground-control points; determination of relative model parameters from image correspondences between two images; and scene reconstruction given image correspondences and ground-control points. The linear pushbroom model leads to theoretical insights that are approximately valid for the full model as well. The epipolar geometry of linear pushbroom cameras is investigated and shown to be totally different from that of a perspective camera. Nevertheless, a matrix analogous to the fundamental matrix of perspective cameras is shown to exist for linear pushbroom sensors. From this it is shown that a scene is determined up to an affine transformation from two views with linear pushbroom cameras.
---
paper_title: Mosaicing New Views: The Crossed-Slits Projection

paper_content:

We introduce anew kind of mosaicing, where the position of the sampling strip varies as a function of the input camera location. The new images that are generated this way correspond to a new projection model defined by two slits, termed here the Crossed-Slits (X-Slits) projection. In this projection model, every 3D point is projected by a ray defined as the line that passes through that point and intersects the two slits. The intersection of the projection rays with the imaging surface defines the image. X-Slits mosaicing provides two benefits. First, the generated mosaics are closer to perspective images than traditional pushbroom mosaics. Second, by simple manipulations of the strip sampling function, we can change the location of one of the virtual slits, providing a virtual walkthrough of a X-Slits camera; all this can be done without recovering any 3D geometry and without calibration. A number of examples where we translate the virtual camera and change its orientation are given; the examples demonstrate realistic changes in parallax, reflections, and occlusions.
---
paper_title: Stereo with oblique cameras

paper_content:

Mosaics acquired by pushbroom cameras, stereo panoramas, omnivergent mosaics, and spherical mosaics can be viewed as images taken by non-central cameras, i.e. cameras that project along rays that do not all intersect at one point. It has been shown that in order to reduce the correspondence search in mosaics to a one-parametric search along curves, the rays of the non-central cameras have to lie in double ruled epipolar surfaces. In this work, we introduce the oblique stereo geometry, which has nonintersecting double ruled epipolar surfaces. We analyze the configurations of mutually oblique rays that see every point in space. We call such configurations oblique cameras. We argue that oblique cameras are important because they are the most non-central cameras among all cameras. We show that oblique cameras, and the corresponding oblique stereo geometry, exist and give an example of a physically realizable oblique stereo geometry.
---
paper_title: Recovering specular surfaces using curved line images

paper_content:

We present a new shape-from-distortion framework for recovering specular (reflective/refractive) surfaces. While most existing approaches rely on accurate correspondences between 2D pixels and 3D points, we focus on analyzing the curved images of 3D lines which we call curved line images or CLIs. Our approach models CLIs of local reflections or refractions using the recently proposed general linear cameras (GLCs). We first characterize all possible CLIs in a GLC. We show that a 3D line will appear as a conic in any GLC. For a fixed GLC, the conic type is invariant to the position and orientation of the line and is determined by the GLC parameters. Furthermore, CLIs under single reflection/refraction can only be lines or hyperbolas. Based on our new theory, we develop efficient algorithms to use multiple CLIs to recover the GLC camera parameters. We then apply the curvature-GLC theory to derive the Gaussian and mean curvatures from the GLC intrinsics. This leads to a complete distortion-based reconstruction framework. Unlike conventional correspondence-based approaches that are sensitive to image distortions, our approach benefits from the CLI distortions. Finally, we demonstrate applying our framework for recovering curvature fields on both synthetic and real specular surfaces.
---
paper_title: General Linear Cameras

paper_content:

We present a General Linear Camera (GLC) model that unifies many previous camera models into a single representation. The GLC model is capable of describing all perspective (pinhole), orthographic, and many multiperspective (including pushbroom and two-slit) cameras, as well as epipolar plane images. It also includes three new and previously unexplored multiperspective linear cameras. Our GLC model is both general and linear in the sense that, given any vector space where rays are represented as points, it describes all 2D affine subspaces (planes) that can be formed by affine combinations of 3 rays. The incident radiance seen along the rays found on subregions of these 2D affine subspaces are a precise definition of a projected image of a 3D scene. The GLC model also provides an intuitive physical interpretation, which can be used to characterize real imaging systems. Finally, since the GLC model provides a complete description of all 2D affine subspaces, it can be used as a tool for first-order differential analysis of arbitrary (higher-order) multiperspective imaging systems.
---
paper_title: Modelling reflections via multiperspective imaging

paper_content:

We present a novel method for analyzing reflections on arbitrary surfaces. We model reflections using a broader than usual class of imaging models, which include both perspective and multiperspective camera types. We provide an analytical framework to locally model reflections as specific multiperspective cameras around every ray based on a new theory of general linear cameras. Our framework better characterizes the complicated image distortions seen on irregular mirror surfaces as well as the conventional catadioptric mirrors. We show the connection between multiperspective camera models and caustic surfaces of reflections and demonstrate how they reveal important surface rulings of the caustics. Finally, we show how to use our analysis to assist mirror design and characterize distortions seen in catadioptric imaging systems.
---
paper_title: Focal Surfaces of Discrete Geometry

paper_content:

The differential geometry of smooth three-dimensional surfaces can be interpreted from one of two perspectives: in terms of oriented frames located on the surface, or in terms of a pair of associated focal surfaces. These focal surfaces are swept by the loci of the principal curvatures' radii. In this article, we develop a focal-surfacebased differential geometry interpretation for discrete mesh surfaces. Focal surfaces have many useful properties. For instance, the normal of each focal surface indicates a principal direction of the corresponding point on the original surface. We provide algorithms to robustly approximate the focal surfaces of a triangle mesh with known or estimated normals. Our approach locally parameterizes the surface normals about a point by their intersections with a pair of parallel planes. We show neighboring normal triplets are constrained to pass simultaneously through two slits, which are parallel to the specified parametrization planes and rule the focal surfaces. We develop both CPU and GPU-based algorithms to efficiently approximate these two slits and, hence, the focal meshes. Our focal mesh estimation also provides a novel discrete shape operator that simultaneously estimates the principal curvatures and principal directions.
---
paper_title: Restricted delaunay triangulations and normal cycle

paper_content:

We address the problem of curvature estimation from sampled smooth surfaces. Building upon the theory of normal cycles, we derive a definition of the curvature tensor for polyhedral surfaces. This definition consists in a very simple and new formula. When applied to a polyhedral approximation of a smooth surface, it yields an efficient and reliable curvature estimation algorithm. Moreover, we bound the difference between the estimated curvature and the one of the smooth surface in the case of restricted Delaunay triangulations.
---
paper_title: Discrete Differential-Geometry Operators for Triangulated 2-Manifolds

paper_content:

This paper proposes a unified and consistent set of flexible tools to approximate important geometric attributes, including normal vectors and curvatures on arbitrary triangle meshes. We present a consistent derivation of these first and second order differential properties using averaging Voronoi cells and the mixed Finite-Element/Finite-Volume method, and compare them to existing formulations. Building upon previous work in discrete geometry, these operators are closely related to the continuous case, guaranteeing an appropriate extension from the continuous to the discrete setting: they respect most intrinsic properties of the continuous differential operators. We show that these estimates are optimal in accuracy under mild smoothness conditions, and demonstrate their numerical quality. We also present applications of these operators, such as mesh smoothing, enhancement, and quality checking, and show results of denoising in higher dimensions, such as for tensor images.
---
paper_title: A theory of multi-perspective defocusing

paper_content:

We present a novel theory for characterizing defocus blurs in multi-perspective cameras such as catadioptric mirrors. Our approach studies how multi-perspective ray geometry transforms under the thin lens. We first use the General Linear Cameras (GLCs) [21] to approximate the incident multi-perspective rays to the lens and then apply a Thin Lens Operator (TLO) to map an incident GLC to the exit GLC. To study defocus blurs caused by the GLC rays, we further introduce a new Ray Spread Function (RSF) model analogous the Point Spread Function (PSF). While PSF models defocus blurs caused by a 3D scene point, RSF models blurs spread by rays. We derive closed form RSFs for incident GLC rays, and we show that for catadioptric cameras with a circular aperture, the RSF can be effectively approximated as a single or mixtures of elliptic-shaped kernels. We apply our method for predicting defocus blurs on commonly used catadioptric cameras and for reducing de-focus blurs in catadioptric projections. Experiments on synthetic and real data demonstrate the accuracy and general applicability of our approach.
---
paper_title: The occlusion camera

paper_content:

We introduce the occlusion camera: a non-pinhole camera with 3D distorted rays. Some of the rays sample surfaces that are occluded in the reference view, while the rest sample visible surfaces. The extra samples alleviate disocclusion errors. The silhouette curves are pushed back, so nearly visible samples become visible. A single occlusion camera covers the entire silhouette of an object, whereas many depth images are required to achieve the same effect. Like regular depth images, occlusion-camera images have a single layer thus the number of samples they contain is bounded by the image resolution, and connectivity is defined implicitly. We construct and use occlusion-camera images in hardware. An occlusion-camera image does not guarantee that all disocclusion errors are avoided. Objects with complex geometry are rendered using the union of the samples stored by a planar pinhole camera and an occlusion camera depth image.
---
paper_title: A theory of multi-perspective defocusing

paper_content:

We present a novel theory for characterizing defocus blurs in multi-perspective cameras such as catadioptric mirrors. Our approach studies how multi-perspective ray geometry transforms under the thin lens. We first use the General Linear Cameras (GLCs) [21] to approximate the incident multi-perspective rays to the lens and then apply a Thin Lens Operator (TLO) to map an incident GLC to the exit GLC. To study defocus blurs caused by the GLC rays, we further introduce a new Ray Spread Function (RSF) model analogous the Point Spread Function (PSF). While PSF models defocus blurs caused by a 3D scene point, RSF models blurs spread by rays. We derive closed form RSFs for incident GLC rays, and we show that for catadioptric cameras with a circular aperture, the RSF can be effectively approximated as a single or mixtures of elliptic-shaped kernels. We apply our method for predicting defocus blurs on commonly used catadioptric cameras and for reducing de-focus blurs in catadioptric projections. Experiments on synthetic and real data demonstrate the accuracy and general applicability of our approach.
---
paper_title: The graph camera

paper_content:

A conventional pinhole camera captures only a small fraction of a 3-D scene due to occlusions. We introduce the graph camera, a non-pinhole with rays that circumvent occluders to create a single layer image that shows simultaneously several regions of interest in a 3-D scene. The graph camera image exhibits good continuity and little redundancy. The graph camera model is literally a graph of tens of planar pinhole cameras. A fast projection operation allows rendering in feed-forward fashion, at interactive rates, which provides support for dynamic scenes. The graph camera is an infrastructure level tool with many applications. We explore the graph camera benefits in the contexts of virtual 3-D scene exploration and summarization, and in the context of real-world 3-D scene visualization. The graph camera allows integrating multiple video feeds seamlessly, which enables monitoring complex real-world spaces with a single image.
---
paper_title: Construction of panoramic image mosaics with global and local alignment

paper_content:

The automatic construction of large, high-resolution image mosaics is an active area of research in the fields of photogrammetry, computer vision, image processing, and computer graphics. Image mosaics can be used for many different applications [163, 1.22]. The most traditional application is the construction of large aerial and satellite photographs from collections of images [186]. More recent applications include scene stabilization and change detection [93], video compression [125, 122, 167] and video indexing [240], increasing the field of view [105, 177, 266] and resolution [126, 50] of a camera, and even simple photo editing [38]. A particularly popular application is the emulation of traditional film-based panoramic photography [175] with digital panoramic mosaics, for applications such as the construction of virtual environments [181, 267] and virtual travel [49].
---
paper_title: Mosaicing New Views: The Crossed-Slits Projection

paper_content:

We introduce anew kind of mosaicing, where the position of the sampling strip varies as a function of the input camera location. The new images that are generated this way correspond to a new projection model defined by two slits, termed here the Crossed-Slits (X-Slits) projection. In this projection model, every 3D point is projected by a ray defined as the line that passes through that point and intersects the two slits. The intersection of the projection rays with the imaging surface defines the image. X-Slits mosaicing provides two benefits. First, the generated mosaics are closer to perspective images than traditional pushbroom mosaics. Second, by simple manipulations of the strip sampling function, we can change the location of one of the virtual slits, providing a virtual walkthrough of a X-Slits camera; all this can be done without recovering any 3D geometry and without calibration. A number of examples where we translate the virtual camera and change its orientation are given; the examples demonstrate realistic changes in parallax, reflections, and occlusions.
---
paper_title: Multiperspective panoramas for cel animation

paper_content:

We describe a new approach for simulating apparent camera motion through a 3D environment. The approach is motivated by a traditional technique used in 2D cel animation, in which a single background image, which we call a multiperspective panorama, is used to incorporate multiple views of a 3D environment as seen from along a given camera path. When viewed through a small moving window, the panorama produces the illusion of 3D motion. In this paper, we explore how such panoramas can be designed by computer, and we examine their application to cel animation in particular. Multiperspective panoramas should also be useful for any application in which predefined camera moves are applied to 3D scenes, including virtual reality fly-throughs, computer games, and architectural walk-throughs. CR Categories: I.3.3 [Computer Graphics]: Picture/Image Generation. Additional
---
paper_title: Caustics of catadioptric cameras

paper_content:

Conventional vision systems and algorithms assume the camera to have a single viewpoint. However, sensors need not always maintain a single viewpoint. For instance, an incorrectly aligned system could cause non-single viewpoints. Also, systems could be designed to specifically deviate from a single viewpoint to trade-off image characteristics such as resolution and field of view. In these cases, the locus of viewpoints forms what is called a caustic. In this paper, we present an in-depth analysis of caustics of catadioptric cameras with conic reflectors. Properties of caustics with respect to field of view and resolution are presented. Finally, we present ways to calibrate conic catadioptric systems and estimate their caustics from known camera motion.
---
paper_title: Modelling reflections via multiperspective imaging

paper_content:

We present a novel method for analyzing reflections on arbitrary surfaces. We model reflections using a broader than usual class of imaging models, which include both perspective and multiperspective camera types. We provide an analytical framework to locally model reflections as specific multiperspective cameras around every ray based on a new theory of general linear cameras. Our framework better characterizes the complicated image distortions seen on irregular mirror surfaces as well as the conventional catadioptric mirrors. We show the connection between multiperspective camera models and caustic surfaces of reflections and demonstrate how they reveal important surface rulings of the caustics. Finally, we show how to use our analysis to assist mirror design and characterize distortions seen in catadioptric imaging systems.
---
paper_title: Mosaicing on adaptive manifolds

paper_content:

Image mosaicing is commonly used to increase the visual field of view by pasting together many images or video frames. Existing mosaicing methods are based on projecting all images onto a predetermined single manifold: A plane is commonly used for a camera translating sideways, a cylinder is used for a panning camera, and a sphere is used for a camera which is both panning and tilting. While different mosaicing methods should therefore be used for different types of camera motion, more general types of camera motion, such as forward motion, are practically impossible for traditional mosaicing. A new methodology to allow image mosaicing in more general cases of camera motion is presented. Mosaicing is performed by projecting thin strips from the images onto manifolds which are adapted to the camera motion. While the limitations of existing mosaicing techniques are a result of using predetermined manifolds, the use of more general manifolds overcomes these limitations.
---
paper_title: The Space of All Stereo Images

paper_content:

A theory of stereo image formation is presented that enables a complete classification of all possible stereo views, including non-perspective varieties. Towards this end, the notion of epipolar geometry is generalized to apply to multiperspective images. It is shown that any stereo pair must consist of rays lying on one of three varieties of quadric surfaces. A unified representation is developed to model all classes of stereo views, based on the concept of a quadric view. The benefits include a unified treatment of projection and triangulation operations for all stereo views. The framework is applied to derive new types of stereo image representations with unusual and useful properties. Experimental examples of these images are constructed and used to obtain 3D binocular object reconstructions.
---
paper_title: Photographing long scenes with multi-viewpoint panoramas

paper_content:

We present a system for producing multi-viewpoint panoramas of long, roughly planar scenes, such as the facades of buildings along a city street, from a relatively sparse set of photographs captured with a handheld still camera that is moved along the scene. Our work is a significant departure from previous methods for creating multi-viewpoint panoramas, which composite thin vertical strips from a video sequence captured by a translating video camera, in that the resulting panoramas are composed of relatively large regions of ordinary perspective. In our system, the only user input required beyond capturing the photographs themselves is to identify the dominant plane of the photographed scene; our system then computes a panorama automatically using Markov Random Field optimization. Users may exert additional control over the appearance of the result by drawing rough strokes that indicate various high-level goals. We demonstrate the results of our system on several scenes, including urban streets, a river bank, and a grocery store aisle.
---
paper_title: Mosaicing New Views: The Crossed-Slits Projection

paper_content:

We introduce anew kind of mosaicing, where the position of the sampling strip varies as a function of the input camera location. The new images that are generated this way correspond to a new projection model defined by two slits, termed here the Crossed-Slits (X-Slits) projection. In this projection model, every 3D point is projected by a ray defined as the line that passes through that point and intersects the two slits. The intersection of the projection rays with the imaging surface defines the image. X-Slits mosaicing provides two benefits. First, the generated mosaics are closer to perspective images than traditional pushbroom mosaics. Second, by simple manipulations of the strip sampling function, we can change the location of one of the virtual slits, providing a virtual walkthrough of a X-Slits camera; all this can be done without recovering any 3D geometry and without calibration. A number of examples where we translate the virtual camera and change its orientation are given; the examples demonstrate realistic changes in parallax, reflections, and occlusions.
---
paper_title: The occlusion camera

paper_content:

We introduce the occlusion camera: a non-pinhole camera with 3D distorted rays. Some of the rays sample surfaces that are occluded in the reference view, while the rest sample visible surfaces. The extra samples alleviate disocclusion errors. The silhouette curves are pushed back, so nearly visible samples become visible. A single occlusion camera covers the entire silhouette of an object, whereas many depth images are required to achieve the same effect. Like regular depth images, occlusion-camera images have a single layer thus the number of samples they contain is bounded by the image resolution, and connectivity is defined implicitly. We construct and use occlusion-camera images in hardware. An occlusion-camera image does not guarantee that all disocclusion errors are avoided. Objects with complex geometry are rendered using the union of the samples stored by a planar pinhole camera and an occlusion camera depth image.
---
paper_title: A rendering framework for multiscale views of 3D models

paper_content:

Images that seamlessly combine views at different levels of detail are appealing. However, creating such multiscale images is not a trivial task, and most such illustrations are handcrafted by skilled artists. This paper presents a framework for direct multiscale rendering of geometric and volumetric models. The basis of our approach is a set of non-linearly bent camera rays that smoothly cast through multiple scales. We show that by properly setting up a sequence of conventional pinhole cameras to capture features of interest at different scales, along with image masks specifying the regions of interest for each scale on the projection plane, our rendering framework can generate non-linear sampling rays that smoothly project objects in a scene at multiple levels of detail onto a single image. We address two important issues with non-linear camera projection. First, our streamline-based ray generation algorithm avoids undesired camera ray intersections, which often result in unexpected images. Second, in order to maintain camera ray coherence and preserve aesthetic quality, we create an interpolated 3D field that defines the contribution of each pinhole camera for determining ray orientations. The resulting multiscale camera has three main applications: (1) presenting hierarchical structure in a compact and continuous manner, (2) achieving focus+context visualization, and (3) creating fascinating and artistic images.
---
paper_title: Geometry of Two-Slit Camera

paper_content:

We analyze the geometry of the two-slit camera and come to two conclusions. First, we show that the definition given by et al. [9] makes sense only if the two slits are not intersecting. Secondly, we prove that the complete image from a two-slit camera cannot be obtained as an intersection of the rays of the two-slit camera with a plane in space. Motivated by the quest for a unified representation of various cameras by simple geometrical objects, we give a new definition of linear oblique cameras as those which comprise all real lines incident with some non-real line and show that it is equivalent with the definition we gave earlier. We also show that no single line neither in the real projective space nor in its coplexification, can be used to define analogously a two-slit camera.
---
paper_title: Stereo panorama with a single camera

paper_content:

Full panoramic images, covering 360 degrees, can be created either by using panoramic cameras or by mosaicing together many regular images. Creating panoramic views in stereo, where one panorama is generated for the left eye, and another panorama is generated for the right eye is more problematic. Earlier attempts to mosaic images from a rotating pair of stereo cameras faced severe problems of parallax and of scale changes. A new family of multiple viewpoint image projections, the Circular Projections, is developed. Two panoramic images taken using such projections can serve as a panoramic stereo pair. A system is described to generates a stereo panoramic image using circular projections from images or video taken by a single rotating camera. The system works in real-time on a PC. It should be noted that the stereo images are created without computation of 3D structure, and the depth effects are created only in the viewer's brain.
---
paper_title: On the epipolar geometry of the Crossed-Slits projection

paper_content:

The Crossed-Slits (X-Slits) camera is defined by two nonintersecting slits, which replace the pinhole in the common perspective camera. Each point in space is projected to the image plane by a ray which passes through the point and the two slits. The X-Slits projection model includes the pushbroom camera as a special case. In addition, it describes a certain class of panoramic images, which are generated from sequences obtained by translating pinhole cameras. In this paper we develop the epipolar geometry of the X-Slits projection model. We show an object which is similar to the fundamental matrix; our matrix, however, describes a quadratic relation between corresponding image points (using the Veronese mapping). Similarly the equivalent of epipolar lines are conics in the image plane. Unlike the pin-hole case, epipolar surfaces do not usually exist in the sense that matching epipolar lines lie on a single surface; we analyze the cases when epipolar surfaces exist, and characterize their properties. Finally, we demonstrate the matching of points in pairs of X-Slits panoramic images.
---
paper_title: Recovering specular surfaces using curved line images

paper_content:

We present a new shape-from-distortion framework for recovering specular (reflective/refractive) surfaces. While most existing approaches rely on accurate correspondences between 2D pixels and 3D points, we focus on analyzing the curved images of 3D lines which we call curved line images or CLIs. Our approach models CLIs of local reflections or refractions using the recently proposed general linear cameras (GLCs). We first characterize all possible CLIs in a GLC. We show that a 3D line will appear as a conic in any GLC. For a fixed GLC, the conic type is invariant to the position and orientation of the line and is determined by the GLC parameters. Furthermore, CLIs under single reflection/refraction can only be lines or hyperbolas. Based on our new theory, we develop efficient algorithms to use multiple CLIs to recover the GLC camera parameters. We then apply the curvature-GLC theory to derive the Gaussian and mean curvatures from the GLC intrinsics. This leads to a complete distortion-based reconstruction framework. Unlike conventional correspondence-based approaches that are sensitive to image distortions, our approach benefits from the CLI distortions. Finally, we demonstrate applying our framework for recovering curvature fields on both synthetic and real specular surfaces.
---
paper_title: Epsilon Stereo Pairs

paper_content:

Human stereo vision works by fusing a pair of perspective images with a purely horizontal parallax. Recent developments suggest that very few varieties of multiperspective stereo pairs exist. In this paper, we introduce a new stereo model, which we call epsilon stereo pairs, for fusing a broader class of multiperspective images. An epsilon stereo pair consists of two images with a slight vertical parallax. We show many multiperspective camera pairs that do not satisfy the stereo constraint can still form epsilon stereo pairs. We then introduce a new ray-space warping algorithm to minimize stereo inconsistencies in an epsilon pair using multiperspective collineations. This makes epsilon stereo model a promising tool for synthesizing close-to-stereo fusions from many non-stereo pairs.
---
paper_title: The Space of All Stereo Images

paper_content:

A theory of stereo image formation is presented that enables a complete classification of all possible stereo views, including non-perspective varieties. Towards this end, the notion of epipolar geometry is generalized to apply to multiperspective images. It is shown that any stereo pair must consist of rays lying on one of three varieties of quadric surfaces. A unified representation is developed to model all classes of stereo views, based on the concept of a quadric view. The benefits include a unified treatment of projection and triangulation operations for all stereo views. The framework is applied to derive new types of stereo image representations with unusual and useful properties. Experimental examples of these images are constructed and used to obtain 3D binocular object reconstructions.
---
paper_title: Recovering shape characteristics on near-flat specular surfaces

paper_content:

We consider the problem of capturing shape characteristics on specular (refractive and reflective) surfaces that are nearly flat. These surfaces are difficult to model using traditional methods based on reconstructing the surface positions and normals. These lower-order shape attributes provide little information to identify important surface characteristics related to distortions. In this paper, we present a framework for recovering the higher-order geometry attributes of specular surfaces. Our method models local reflections and refractions in terms of a special class of multiperspective cameras called the general linear cameras (GLCs). We then develop a new theory that correlates the higher-order differential geometry attributes with the local GLCs. Specifically, we show that Gaussian and mean curvature can be directly derived from the camera intrinsics of the local GLCs. We validate this theory on both synthetic and real-world specular surfaces. Our method places a known pattern in front of a reflective surface or beneath a refractive surface and captures a distorted image on the surface. We then compute the optimal GLC using a sparse set of correspondences and recover the curvatures from the GLC. Experiments demonstrate that our methods are robust and highly accurate.
---
paper_title: Catadioptric omnidirectional camera

paper_content:

Conventional video cameras have limited fields of view that make them restrictive in a variety of vision applications. There are several ways to enhance the field of view of an imaging system. However, the entire imaging system must have a single effective viewpoint to enable the generation of pure perspective images from a sensed image. A new camera with a hemispherical field of view is presented. Two such cameras can be placed back-to-back, without violating the single viewpoint constraint, to arrive at a truly omnidirectional sensor. Results are presented on the software generation of pure perspective images from an omnidirectional image, given any user-selected viewing direction and magnification. The paper concludes with a discussion on the spatial resolution of the proposed camera.
---
paper_title: A Theory of Single-Viewpoint Catadioptric Image Formation

paper_content:

Conventional video cameras have limited fields of view which make them restrictive for certain applications in computational vision. A catadioptric sensor uses a combination of lenses and mirrors placed in a carefully arranged configuration to capture a much wider field of view. One important design goal for catadioptric sensors is choosing the shapes of the mirrors in a way that ensures that the complete catadioptric system has a single effective viewpoint. The reason a single viewpoint is so desirable is that it is a requirement for the generation of pure perspective images from the sensed images. In this paper, we derive the complete class of single-lens single-mirror catadioptric sensors that have a single viewpoint. We describe all of the solutions in detail, including the degenerate ones, with reference to many of the catadioptric systems that have been proposed in the literature. In addition, we derive a simple expression for the spatial resolution of a catadioptric sensor in terms of the resolution of the cameras used to construct it. Moreover, we include detailed analysis of the defocus blur caused by the use of a curved mirror in a catadioptric sensor.
---
paper_title: Caustics of catadioptric cameras

paper_content:

Conventional vision systems and algorithms assume the camera to have a single viewpoint. However, sensors need not always maintain a single viewpoint. For instance, an incorrectly aligned system could cause non-single viewpoints. Also, systems could be designed to specifically deviate from a single viewpoint to trade-off image characteristics such as resolution and field of view. In these cases, the locus of viewpoints forms what is called a caustic. In this paper, we present an in-depth analysis of caustics of catadioptric cameras with conic reflectors. Properties of caustics with respect to field of view and resolution are presented. Finally, we present ways to calibrate conic catadioptric systems and estimate their caustics from known camera motion.
---
paper_title: Modelling reflections via multiperspective imaging

paper_content:

We present a novel method for analyzing reflections on arbitrary surfaces. We model reflections using a broader than usual class of imaging models, which include both perspective and multiperspective camera types. We provide an analytical framework to locally model reflections as specific multiperspective cameras around every ray based on a new theory of general linear cameras. Our framework better characterizes the complicated image distortions seen on irregular mirror surfaces as well as the conventional catadioptric mirrors. We show the connection between multiperspective camera models and caustic surfaces of reflections and demonstrate how they reveal important surface rulings of the caustics. Finally, we show how to use our analysis to assist mirror design and characterize distortions seen in catadioptric imaging systems.
---
paper_title: Non-Single Viewpoint Catadioptric Cameras: Geometry and Analysis

paper_content:

Abstract.Conventional vision systems and algorithms assume the imaging system to have a single viewpoint. However, these imaging systems need not always maintain a single viewpoint. For instance, an incorrectly aligned catadioptric system could cause non-single viewpoints. Moreover, a lot of flexibility in imaging system design can be achieved by relaxing the need for imaging systems to have a single viewpoint. Thus, imaging systems with non-single viewpoints can be designed for specific imaging tasks, or image characteristics such as field of view and resolution. The viewpoint locus of such imaging systems is called a caustic.In this paper, we present an in-depth analysis of caustics of catadioptric cameras with conic reflectors. We use a simple parametric model for both, the reflector and the imaging system, to derive an analytic solution for the caustic surface. This model completely describes the imaging system and provides a map from pixels in the image to their corresponding viewpoints and viewing direction. We use the model to analyze the imaging system's properties such as field of view, resolution and other geometric properties of the caustic itself. In addition, we present a simple technique to calibrate the class of conic catadioptric cameras and estimate their caustics from known camera motion. The analysis and results we present in this paper are general and can be applied to any catadioptric imaging system whose reflector has a parametric form.
---
paper_title: Multiperspective projection and collineation

paper_content:

We present theories of multiperspective projection and collineation. Given an arbitrary multiperspective imaging system that captures smoothly varying set of rays, we show how to map the rays onto a 2D ray manifold embedded in a 4D linear vector space. The characteristics of this imaging system, such as its projection, collineation, and image distortions can be analyzed by studying the 2D tangent planes of this ray manifold. These tangent planes correspond to the recently proposed general linear camera (GLC) model. In this paper, we study the imaging process of the GLCs. We show the GLC imaging process can be broken down into two separate stages: the mapping of 3D geometry to rays and the sampling of those rays over an image plane. We derive a closed-form solution to projecting 3D points in a scene to rays in a GLC. A GLC image is created by sampling these rays over an image plane. We develop a notion of GLC collineation analogous to pinhole cameras. GLC collineation describes the transformation between the images of a single GLC due to changes in sampling and image plane selection. We show that general GLC collineations can be characterized by a quartic (4th order) rational function. GLC projection and collineation provides a basis for developing new computer vision algorithms suitable for analyzing a wider range of imaging systems than current methods, based on simple pinhole projection models, permit.
---
paper_title: Multiview radial catadioptric imaging for scene capture

paper_content:

In this paper, we present a class of imaging systems, called radial imaging systems, that capture a scene from a large number of view-points within a single image, using a camera and a curved mirror. These systems can recover scene properties such as geometry, reflectance, and texture. We derive analytic expressions that describe the properties of a complete family of radial imaging systems, including their loci of viewpoints, fields of view, and resolution characteristics. We have built radial imaging systems that, from a single image, recover the frontal 3D structure of an object, generate the complete texture map of a convex object, and estimate the parameters of an analytic BRDF model for an isotropic material. In addition, one of our systems can recover the complete geometry of a convex object by capturing only two images. These results show that radial imaging systems are simple, effective, and convenient devices for a wide range of applications in computer graphics and computer vision.
---
paper_title: Axial-cones: modeling spherical catadioptric cameras for wide-angle light field rendering

paper_content:

Catadioptric imaging systems are commonly used for wide-angle imaging, but lead to multi-perspective images which do not allow algorithms designed for perspective cameras to be used. Efficient use of such systems requires accurate geometric ray modeling as well as fast algorithms. We present accurate geometric modeling of the multi-perspective photo captured with a spherical catadioptric imaging system using axial-cone cameras: multiple perspective cameras lying on an axis each with a different viewpoint and a different cone of rays. This modeling avoids geometric approximations and allows several algorithms developed for perspective cameras to be applied to multi-perspective catadioptric cameras. ::: We demonstrate axial-cone modeling in the context of rendering wide-angle light fields, captured using a spherical mirror array. We present several applications such as spherical distortion correction, digital refocusing for artistic depth of field effects in wide-angle scenes, and wide-angle dense depth estimation. Our GPU implementation using axial-cone modeling achieves up to three orders of magnitude speed up over ray tracing for these applications.
---
paper_title: Theory and Calibration for Axial Cameras

paper_content:

Although most works in computer vision use perspective or other central cameras, the interest in non-central camera models has increased lately, especially with respect to omnidirectional vision. Calibration and structure-from-motion algorithms exist for both, central and non-central cameras. An intermediate class of cameras, although encountered rather frequently, has received less attention. So-called axial cameras are non-central but their projection rays are constrained by the existence of a line that cuts all of them. This is the case for stereo systems, many non-central catadioptric cameras and pushbroom cameras for example. In this paper, we study the geometry of axial cameras and propose a calibration approach for them. We also describe the various axial catadioptric configurations which are more common and less restrictive than central catadioptric ones. Finally we used simulations and real experiments to prove the validity of our theory.
---
paper_title: The Space of All Stereo Images

paper_content:

A theory of stereo image formation is presented that enables a complete classification of all possible stereo views, including non-perspective varieties. Towards this end, the notion of epipolar geometry is generalized to apply to multiperspective images. It is shown that any stereo pair must consist of rays lying on one of three varieties of quadric surfaces. A unified representation is developed to model all classes of stereo views, based on the concept of a quadric view. The benefits include a unified treatment of projection and triangulation operations for all stereo views. The framework is applied to derive new types of stereo image representations with unusual and useful properties. Experimental examples of these images are constructed and used to obtain 3D binocular object reconstructions.
---
paper_title: Hand-held Schlieren Photography with Light Field probes

paper_content:

We introduce a new approach to capturing refraction in transparent media, which we call Light Field Background Oriented Schlieren Photography (LFBOS). By optically coding the locations and directions of light rays emerging from a light field probe, we can capture changes of the refractive index field between the probe and a camera or an observer. Rather than using complicated and expensive optical setups as in traditional Schlieren photography we employ commodity hardware; our prototype consists of a camera and a lenslet array. By carefully encoding the color and intensity variations of a 4D probe instead of a diffuse 2D background, we avoid expensive computational processing of the captured data, which is necessary for Background Oriented Schlieren imaging (BOS). We analyze the benefits and limitations of our approach and discuss application scenarios.
---
paper_title: Angular domain reconstruction of dynamic 3D fluid surfaces

paper_content:

We present a novel and simple computational imaging solution to robustly and accurately recover 3D dynamic fluid surfaces. Traditional specular surface reconstruction schemes place special patterns (checkerboard or color patterns) beneath the fluid surface to establish point-pixel correspondences. However, point-pixel correspondences alone are insufficient to recover surface normal or height and they rely on additional constraints to resolve the ambiguity. In this paper, we exploit using Bokode — a computational optical device that emulates a pinhole projector — for capturing ray-ray correspondences which can then be used to directly recover the surface normals. We further develop a robust feature matching algorithm based on the Active-Appearance Model to robustly establishing ray-ray correspondences. Our solution results in an angularly sampled normal field and we derive a new angular-domain surface integration scheme to recover the surface from the normal fields. Specifically, we reformulate the problem as an over-constrained linear system under spherical coordinate and solve it using Singular Value Decomposition. Experiments results on real and synthetic surfaces demonstrate that our approach is robust and accurate, and is easier to implement than state-of-the-art multi-camera based approaches.
---
paper_title: Axial-cones: modeling spherical catadioptric cameras for wide-angle light field rendering

paper_content:

Catadioptric imaging systems are commonly used for wide-angle imaging, but lead to multi-perspective images which do not allow algorithms designed for perspective cameras to be used. Efficient use of such systems requires accurate geometric ray modeling as well as fast algorithms. We present accurate geometric modeling of the multi-perspective photo captured with a spherical catadioptric imaging system using axial-cone cameras: multiple perspective cameras lying on an axis each with a different viewpoint and a different cone of rays. This modeling avoids geometric approximations and allows several algorithms developed for perspective cameras to be applied to multi-perspective catadioptric cameras. ::: We demonstrate axial-cone modeling in the context of rendering wide-angle light fields, captured using a spherical mirror array. We present several applications such as spherical distortion correction, digital refocusing for artistic depth of field effects in wide-angle scenes, and wide-angle dense depth estimation. Our GPU implementation using axial-cone modeling achieves up to three orders of magnitude speed up over ray tracing for these applications.
---
paper_title: Fourier slice photography

paper_content:

This paper contributes to the theory of photograph formation from light fields. The main result is a theorem that, in the Fourier domain, a photograph formed by a full lens aperture is a 2D slice in the 4D light field. Photographs focused at different depths correspond to slices at different trajectories in the 4D space. The paper demonstrates the utility of this theorem in two different ways. First, the theorem is used to analyze the performance of digital refocusing, where one computes photographs focused at different depths from a single light field. The analysis shows in closed form that the sharpness of refocused photographs increases linearly with directional resolution. Second, the theorem yields a Fourier-domain algorithm for digital refocusing, where we extract the appropriate 2D slice of the light field's Fourier transform, and perform an inverse 2D Fourier transform. This method is faster than previous approaches.
---
paper_title: Bokode: imperceptible visual tags for camera based interaction from a distance

paper_content:

We show a new camera based interaction solution where an ordinary camera can detect small optical tags from a relatively large distance. Current optical tags, such as barcodes, must be read within a short range and the codes occupy valuable physical space on products. We present a new low-cost optical design so that the tags can be shrunk to 3mm visible diameter, and unmodified ordinary cameras several meters away can be set up to decode the identity plus the relative distance and angle. The design exploits the bokeh effect of ordinary cameras lenses, which maps rays exiting from an out of focus scene point into a disk like blur on the camera sensor. This bokeh-code or Bokode is a barcode design with a simple lenslet over the pattern. We show that a code with 15μm features can be read using an off-the-shelf camera from distances of up to 2 meters. We use intelligent binary coding to estimate the relative distance and angle to the camera, and show potential for applications in augmented reality and motion capture. We analyze the constraints and performance of the optical system, and discuss several plausible application scenarios.
---
paper_title: The focused plenoptic camera

paper_content:

Plenoptic cameras, constructed with internal microlens arrays, focus those microlenses at infinity in order to sample the 4D radiance directly at the microlenses. The consequent assumption is that each microlens image is completely defocused with respect to to the image created by the main camera lens and the outside object. As a result, only a single pixel in the final image can be rendered from it, resulting in disappointingly low resolution. In this paper, we present a new approach to lightfield capture and image rendering that interprets the microlens array as an imaging system focused on the focal plane of the main camera lens. This approach captures a lightfield with significantly higher spatial resolution than the traditional approach, allowing us to render high resolution images that meet the expectations of modern photographers. Although the new approach samples the lightfield with reduced angular density, analysis and experimental results demonstrate that there is sufficient parallax to completely support lightfield manipulation algorithms such as refocusing and novel views
---
paper_title: Dynamically reparameterized light fields

paper_content:

This research further develops the light field and lumigraph image-based rendering methods and extends their utility. We present alternate parameterizations that permit 1) interactive rendering of moderately sampled light fields of scenes with significant, unknown depth variation and 2) low-cost, passive autostereoscopic viewing. Using a dynamic reparameterization, these techniques can be used to interactively render photographic effects such as variable focus and depth-of-field within a light field. The dynamic parameterization is independent of scene geometry and does not require actual or approximate geometry of the scene. We explore the frequency domain and ray-space aspects of dynamic reparameterization, and present an interactive rendering technique that takes advantage of today's commodity rendering hardware.
---
paper_title: The Plenoptic Function and the Elements of Early Vision

paper_content:

What are the elements of early vision? This question might be taken to mean, What are the fundamental atoms of vision?—and might be variously answered in terms ofsuch candidate structures as edges, peaks, corners, and so on. In this chapter we adopt a rather different point of view and ask the question, What are the fundamentalsubstances of vision? This distinction is important becausewe wish to focus on the first steps in extraction of visualinformation. At this level it is premature to talk aboutdiscrete objects, even such simple ones as edges and corners.There is general agreement that early vision involvesmeasurements of a number of basic image properties in-cluding orientation, color, motion, and so on. Figure l.lshows a caricature (in the style of Neisser, 1976), of the sort of architecture that has become quite popular as a model for both human and machine vision. The first stageof processing involves a set of parallel pathways, eachdevoted to one particular-visual property. We propose that the measurements of these basic properties be con-sidered as the elements of early vision. We think of earlyvision as measuring the amounts of various kinds of vi-sual "substances" present in the image (e.g., redness orrightward motion energy). In other words, we are inter- ested in how early vision measures “stuff” rather than in how it labels “things.”What, then, are these elementary visual substances?Various lists have been compiled using a mixture of intui-tion and experiment. Electrophysiologists have describedneurons in striate cortex that are selectively sensitive tocertain visual properties; for reviews, see Hubel (1988) and DeValois and DeValois (1988). Psychophysicists haveinferred the existence of channels that are tuned for cer- tain visual properties; for reviews, see Graham (1989), Olzak and Thomas (1986), Pokorny and Smith (1986), and Watson (1986). Researchers in perception have foundaspects of visual stimuli that are processed pre-attentive- ly (Beck, 1966; Bergen & Julesz, 1983; Julesz & Bergen,
---
paper_title: Single Lens Stereo with a Plenoptic Camera

paper_content:

Ordinary cameras gather light across the area of their lens aperture, and the light striking a given subregion of the aperture is structured somewhat differently than the light striking an adjacent subregion. By analyzing this optical structure, one can infer the depths of the objects in the scene, i.e. one can achieve single lens stereo. The authors describe a camera for performing this analysis. It incorporates a single main lens along with a lenticular array placed at the sensor plane. The resulting plenoptic camera provides information about how the scene would look when viewed from a continuum of possible viewpoints bounded by the main lens aperture. Deriving depth information is simpler than in a binocular stereo system because the correspondence problem is minimized. The camera extracts information about both horizontal and vertical parallax, which improves the reliability of the depth estimates. >
---
paper_title: High performance imaging using large camera arrays

paper_content:

The advent of inexpensive digital image sensors and the ability to create photographs that combine information from a number of sensed images are changing the way we think about photography. In this paper, we describe a unique array of 100 custom video cameras that we have built, and we summarize our experiences using this array in a range of imaging applications. Our goal was to explore the capabilities of a system that would be inexpensive to produce in the future. With this in mind, we used simple cameras, lenses, and mountings, and we assumed that processing large numbers of images would eventually be easy and cheap. The applications we have explored include approximating a conventional single center of projection video camera with high performance along one or more axes, such as resolution, dynamic range, frame rate, and/or large aperture, and using multiple cameras to approximate a video camera with a large synthetic aperture. This permits us to capture a video light field, to which we can apply spatiotemporal view interpolation algorithms in order to digitally simulate time dilation and camera motion. It also permits us to create video sequences using custom non-uniform synthetic apertures.
---
paper_title: Multiperspective stereo matching and volumetric reconstruction

paper_content:

Stereo matching and volumetric reconstruction are the most explored 3D scene recovery techniques in computer vision. Many existing approaches assume perspective input images and use the epipolar constraint to reduce the search space and improve the accuracy. In this paper we present a novel framework that uses multi-perspective cameras for stereo matching and volumetric reconstruction. Our approach first decomposes a multi-perspective camera into piecewise primitive General Linear Cameras or GLCs [32]. A pair of GLCs in general do not satisfy the epipolar constraint. However, they still form a nearly stereo pair. We develop a new Graph-Cut-based algorithm to account for the slight vertical parallax using the GLC ray geometry. We show that the recovered pseudo disparity map conveys important depth cues analogous to perspective stereo matching. To more accurately reconstruct a 3D scene, we develop a new multi-perspective volumetric reconstruction method. We discretize the scene into voxels and apply the GLC back-projections to map the voxel onto each input multi-perspective camera. Finally, we apply the graph-cut algorithm to optimize the 3D embedded voxel graph. We demonstrate our algorithms on both synthetic and real multi-perspective cameras. Experimental results show that our methods are robust and reliable.
---
paper_title: Refractive shape from light field distortion

paper_content:

Acquiring transparent, refractive objects is challenging as these kinds of objects can only be observed by analyzing the distortion of reference background patterns. We present a new, single image approach to reconstructing thin transparent surfaces, such as thin solids or surfaces of fluids. Our method is based on observing the distortion of light field background illumination. Light field probes have the potential to encode up to four dimensions in varying colors and intensities: spatial and angular variation on the probe surface; commonly employed reference patterns are only two-dimensional by coding either position or angle on the probe. We show that the additional information can be used to reconstruct refractive surface normals and a sparse set of control points from a single photograph.
---
paper_title: Catadioptric projectors

paper_content:

We present a catadioptric projector analogous to a catadioptric camera by combining a commodity digital projector with additional optical units. We show that, by using specially shaped reflectors/refractors, catadioptric projectors can offer an unprecedented level of flexibility in aspect ratio, size, and field of view. We also present efficient algorithms to reduce projection artifacts in catadioptric projectors, such as distortions, scattering, and defocusing. Instead of recovering the reflector/refractor geometry, our approach directly models the light transport between the projector and the viewpoint using the light transport matrix (LTM). We show how to efficiently approximate the pseudo inverse of the LTM and apply it to find the optimal input image that produces least projection distortions. Furthermore, we present a projection defocus analysis for reflector and thin refractor based catadioptric projectors. We show that defocus blur can be interpreted as spatially-varying Gaussian blurs on the input image. We then measure the kernels directly from the LTM and apply deconvolution to optimize the input image. We demonstrate the practical uses of catadioptric projectors in panoramic and omni-directional projections. Our new system achieves much wider field-of-view projection while maintaining sharpness and low geometric and photometric distortions.
---
paper_title: Capturing and Rendering with Incident Light Fields

paper_content:

This paper presents a process for capturing spatially and directionally varying illumination from a real-world scene and using this lighting to illuminate computer-generated objects. We use two devices for capturing such illumination. In the first we photograph an array of mirrored spheres in high dynamic range to capture the spatially varying illumination. In the second, we obtain higher resolution data by capturing images with an high dynamic range omnidirectional camera as it traverses across a plane. For both methods we apply the light field technique to extrapolate the incident illumination to a volume. We render computer-generated objects as illuminated by this captured illumination using a custom shader within an existing global illumination rendering system. To demonstrate our technique we capture several spatially-varying lighting environments with spotlights, shadows, and dappled lighting and use them to illuminate synthetic scenes. We also show comparisons to real objects under the same illumination.
---
paper_title: Dappled photography: mask enhanced cameras for heterodyned light fields and coded aperture refocusing

paper_content:

We describe a theoretical framework for reversibly modulating 4D light fields using an attenuating mask in the optical path of a lens based camera. Based on this framework, we present a novel design to reconstruct the 4D light field from a 2D camera image without any additional refractive elements as required by previous light field cameras. The patterned mask attenuates light rays inside the camera instead of bending them, and the attenuation recoverably encodes the rays on the 2D sensor. Our mask-equipped camera focuses just as a traditional camera to capture conventional 2D photos at full sensor resolution, but the raw pixel values also hold a modulated 4D light field. The light field can be recovered by rearranging the tiles of the 2D Fourier transform of sensor values into 4D planes, and computing the inverse Fourier transform. In addition, one can also recover the full resolution image information for the in-focus parts of the scene. ::: We also show how a broadband mask placed at the lens enables us to compute refocused images at full sensor resolution for layered Lambertian scenes. This partial encoding of 4D ray-space data enables editing of image contents by depth, yet does not require computational recovery of the complete 4D light field.
---
paper_title: Light field rendering

paper_content:

A number of techniques have been proposed for flying through scenes by redisplaying previously rendered or digitized views. Techniques have also been proposed for interpolating between views by warping input images, using depth information or correspondences between multiple images. In this paper, we describe a simple and robust method for generating new views from arbitrary camera positions without depth information or feature matching, simply by combining and resampling the available images. The key to this technique lies in interpreting the input images as 2D slices of a 4D function the light field. This function completely characterizes the flow of light through unobstructed space in a static scene with fixed illumination. We describe a sampled representation for light fields that allows for both efficient creation and display of inward and outward looking views. We hav e created light fields from large arrays of both rendered and digitized images. The latter are acquired using a video camera mounted on a computer-controlled gantry. Once a light field has been created, new views may be constructed in real time by extracting slices in appropriate directions. Since the success of the method depends on having a high sample rate, we describe a compression system that is able to compress the light fields we have generated by more than a factor of 100:1 with very little loss of fidelity. We also address the issues of antialiasing during creation, and resampling during slice extraction. CR Categories: I.3.2 [Computer Graphics]: Picture/Image Generation — Digitizing and scanning, Viewing algorithms; I.4.2 [Computer Graphics]: Compression — Approximate methods Additional keywords: image-based rendering, light field, holographic stereogram, vector quantization, epipolar analysis
---

Title: Ray Geometry in Non-Pinhole Cameras: A Survey

Section 1: Introduction
Description 1: Introduce the concept of pinhole and non-pinhole cameras, focusing on historical background, basic principles, and the motivation for exploring non-pinhole imaging models.

Section 2: Scope
Description 2: Outline the theoretical and application aspects covered in the survey, including the study of non-pinhole imaging models in ray space and their applications in computer vision, computational photography, and computer graphics.

Section 3: Pinhole optics
Description 3: Discuss the fundamentals of pinhole camera optics, including the geometric principles and the basic imaging process, revisited via ray-space analysis.

Section 4: Ray space
Description 4: Explain the Two-Plane Parametrization (2PP) and its application in representing rays in free space, including the geometric structures of rays in pinhole and orthographic cameras.

Section 5: Pinhole ray geometry
Description 5: Analyze the ray geometry specific to pinhole cameras and orthographic cameras, using pinhole and orthographic ray constraints to derive imaging properties.

Section 6: Slit ray geometry
Description 6: Extend the analysis to more general ray configurations that pass through a line or slit, defining linear and bilinear constraints for parallel and non-parallel slits.

Section 7: The thin lens operator
Description 7: Describe how a thin lens transforms incident rays, introducing the Thin Lens Equation and the Thin Lens Operator (TLO) and their effects on ray geometry.

Section 8: Non-pinhole imaging models
Description 8: Present different types of non-pinhole cameras, including classical non-pinhole cameras (Pushbroom, XSlit) and general non-pinhole cameras analyzed via ray manifolds.

Section 9: General linear cameras (GLC)
Description 9: Discuss the General Linear Camera (GLC) model, its classification, and the ray characteristic equation used to determine the type of non-pinhole camera.

Section 10: Case study 1: reflection on curved mirrors
Description 10: Demonstrate the application of GLC analysis in modeling reflections in catadioptric cameras, characterizing local reflections as different types of GLCs.

Section 11: Case study 2: 3D surfaces
Description 11: Explore the conversion of 3D surfaces into 2D ray manifolds using the normal-ray model and principal curvature analysis.

Section 12: Non-pinhole camera through the thin lens
Description 12: Review the transformation of ray geometry through the thin lens, including GLC transformations and slit-direction dualities.

Section 13: Case study 3: defocus analysis in catadioptric cameras
Description 13: Analyze defocus effects in catadioptric imaging systems using Ray Spread Function (RSF) analysis for complex reflection and projection scenarios.

Section 14: Applications of synthetic non-pinhole cameras
Description 14: Discuss a range of applications where synthetic non-pinhole cameras are used, such as panorama synthesis, non-photorealistic rendering, and multi-perspective effects.

Section 15: Stereo and 3D reconstruction
Description 15: Address extensions of traditional stereo matching algorithms to non-pinhole geometry for 3D reconstruction and shape-from-distortion methods.

Section 16: Real non-pinhole imaging systems
Description 16: Present actual implementations of non-pinhole cameras, including centric and non-centric catadioptric cameras, axial cameras, and catadioptric projectors.

Section 17: Future directions
Description 17: Propose future research directions in non-pinhole camera development, potential new acquisition devices, and advancements in computational and differential geometry for ray space analysis.