IMAGE BASED 3D FACE RECONSTRUCTION: A SURVEY
---
paper_title: A Survey of 3D Face Recognition Methods

paper_content:

Many researches in face recognition have been dealing with the challenge of the great variability in head pose, lighting intensity and direction,facial expression, and aging. The main purpose of this overview is to describe the recent 3D face recognition algorithms. The last few years more and more 2D face recognition algorithms are improved and tested on less than perfect images. However, 3D models hold more information of the face, like surface information, that can be used for face recognition or subject discrimination. Another major advantage is that 3D face recognition is pose invariant. A disadvantage of most presented 3D face recognition methods is that they still treat the human face as a rigid object. This means that the methods aren't capable of handling facial expressions. Although 2D face recognition still seems to outperform the 3D face recognition methods, it is expected that this will change in the near future.
---
paper_title: A survey of approaches and challenges in 3D and multi-modal 3D + 2D face recognition

paper_content:

This survey focuses on recognition performed by matching models of the three-dimensional shape of the face, either alone or in combination with matching corresponding two-dimensional intensity images. Research trends to date are summarized, and challenges confronting the development of more accurate three-dimensional face recognition are identified. These challenges include the need for better sensors, improved recognition algorithms, and more rigorous experimental methodology.
---
paper_title: Facial feature extraction for quick 3D face modeling

paper_content:

There are two main processes to create a 3D animatable facial model from photographs. The first is to extract features such as eyes, nose, mouth and chin curves on the photographs. The second is to create 3D individualized facial model using extracted feature information. The final facial model is expected to have an individualized shape, photograph-realistic skin color, and animatable structures. Here, we describe our novel approach to detect features automatically using a statistical analysis for facial information. We are not only interested in the location of the features but also the shape of local features. How to create 3D models from detected features is also explained and several resulting 3D facial models are illustrated and discussed.
---
paper_title: Automatic creation of 3D facial models

paper_content:

Model-based encoding of human facial features for narrowband visual communication is described. Based on an already prepared 3D human model, this coding method detects and understands a person's body motion and facial expressions. It expresses the essential information as compact codes and transmits it. At the receiving end, this code becomes the basis for modifying the 3D model of the person and thereby generating lifelike human images. The feature extraction used by the system to acquire data for regions or edges that express the eyes, nose, mouth, and outlines of the face and hair is discussed. The way in which the system creates a 3D model of the person by using the features extracted in the first part to modify a generic head model is also discussed. >
---
paper_title: Recovering non-rigid 3D shape from image streams

paper_content:

The paper addresses the problem of recovering 3D non-rigid shape models from image sequences. For example, given a video recording of a talking person, we would like to estimate a 3D model of the lips and the full face and its internal modes of variation. Many solutions that recover 3D shape from 2D image sequences have been proposed; these so-called structure-from-motion techniques usually assume that the 3D object is rigid. For example, C. Tomasi and T. Kanades' (1992) factorization technique is based on a rigid shape matrix, which produces a tracking matrix of rank 3 under orthographic projection. We propose a novel technique based on a non-rigid model, where the 3D shape in each frame is a linear combination of a set of basis shapes. Under this model, the tracking matrix is of higher rank, and can be factored in a three-step process to yield pose, configuration and shape. To the best of our knowledge, this is the first model free approach that can recover from single-view video sequences nonrigid shape models. We demonstrate this new algorithm on several video sequences. We were able to recover 3D non-rigid human face and animal models with high accuracy.
---
paper_title: Stereo matching with energy-minimizing snake grid for 3D face modeling

paper_content:

An energy minimizing snake algorithm that runs over a grid is designed and used to reconstruct high resolution 3D human faces from pairs of stereo images. The accuracy of reconstructed 3D data from stereo depends highly on how well stereo correspondences are established during the feature matching step. Establishing stereo correspondences on human faces is often ill posed and hard to achieve because of uniform texture, slow changes in depth, occlusion, and lack of gradient. We designed an energy minimizing algorithm that accurately finds correspondences on face images despite the aforementioned characteristics. The algorithm helps establish stereo correspondences unambiguously by applying a coarse-to-fine energy minimizing snake in grid format and yields a high resolution reconstruction at nearly every point of the image. Initially, the grid is stabilized using matches at a few selected high confidence edge points. The grid then gradually and consistently spreads over the low gradient regions of the image to reveal the accurate depths of object points. The grid applies its internal energy to approximate mismatches in occluded and noisy regions and to maintain smoothness of the reconstructed surfaces. The grid works in such a way that with every increment in reconstruction resolution, less time is required to establish correspondences. The snake used the curvature of the grid and gradient of image regions to automatically select its energy parameters and approximate the unmatched points using matched points from previous iterations, which also accelerates the overall matching process. The algorithm has been applied for the reconstruction of 3D human faces, and experimental results demonstrate the effectiveness and accuracy of the reconstruction.
---
paper_title: A morphable model for the synthesis of 3D faces

paper_content:

In this paper, a new technique for modeling textured 3D faces is introduced. 3D faces can either be generated automatically from one or more photographs, or modeled directly through an intuitive user interface. Users are assisted in two key problems of computer aided face modeling. First, new face images or new 3D face models can be registered automatically by computing dense one-to-one correspondence to an internal face model. Second, the approach regulates the naturalness of modeled faces avoiding faces with an “unlikely” appearance. Starting from an example set of 3D face models, we derive a morphable face model by transforming the shape and texture of the examples into a vector space representation. New faces and expressions can be modeled by forming linear combinations of the prototypes. Shape and texture constraints derived from the statistics of our example faces are used to guide manual modeling or automated matching algorithms. We show 3D face reconstructions from single images and their applications for photo-realistic image manipulations. We also demonstrate face manipulations according to complex parameters such as gender, fullness of a face or its distinctiveness.
---
paper_title: Shape reconstruction of 3D bilaterally symmetric surfaces

paper_content:

The paper presents a new approach for shape recovery based on integrating geometric and photometric information. We consider 3D objects which are symmetric with respect to a plane (e.g., faces) and their reconstruction from a single image. Both the viewpoint and the illumination are not necessarily frontal. In principle, no correspondence between symmetric points is required, but knowledge about a few corresponding pairs accelerates the process. The basic idea is that an image taken from a general, non-frontal viewpoint, under non-frontal illumination can be regarded as a pair of images of half of the object, taken from two different viewing positions and two different lighting directions. We show that integrating the photometric and geometric information yields the unknown lighting and viewing parameters, as well as dense correspondence between pairs of symmetric points. As a result, a dense shape recovery of the object is computed. The method has been implemented and tested experimentally on simulated and real data.
---
paper_title: Shape from recognition and learning: recovery of 3-D face shapes

paper_content:

In this paper a novel framework for the recovery of 3D surfaces of faces from single images is developed. The underlying principle is shape from recognition, i.e. the idea that pre-recognizing face parts can constrain the space of possible solutions to the image irradiance equation, thus allowing robust recovery of the 3D structure of a specific part. Shape recovery of the recognized part is based on specialized backpropagation based neural networks, each of which is employed in the recovery of a particular face part. Representation using principal components allows to efficiently encode classes of objects such as nose, lips, etc. The specialized networks are designed and trained to map the principal component coefficients of the shading images to another set of principal component coefficients that represent the corresponding 3D surface shapes. A method for integrating recovered 3D surface regions by minimizing the sum squared error in overlapping areas is also derived. Quantitative analysis of the reconstruction of the surface parts show relatively small errors indicating that the method is robust and accurate. The recovery of a complete face is performed by minimal squared error merging efface parts.
---
paper_title: Illumination-insensitive face recognition using symmetric shape-from-shading

paper_content:

Sensitivity to variations in illumination is a fundamental and challenging problem in face recognition. In this paper, we describe a new method based on symmetric shape-from-shading (SSFS) to develop a face recognition system that is robust to changes in illumination. The basic idea of this approach is to use the SSFS algorithm as a tool to obtain a prototype image which is illumination-normalized. It has been shown that the SSFS algorithm has a unique point-wise solution. But it is still difficult to recover accurate shape information given a single real face image with complex shape and varying albedo. In stead, we utilize the fact that all faces share a similar shape making the direct computation of the prototype image from a given face image feasible. Finally, to demonstrate the efficacy of our method, we have applied it to several publicly available face databases.
---
paper_title: Statistical symmetric shape from shading for 3d structure recovery of faces

paper_content:

In this paper, we aim to recover the 3D shape of a human face using a single image. We use a combination of symmetric shape from shading by Zhao and Chellappa and statistical approach for facial shape reconstruction by Atick, Griffin and Redlich. Given a single frontal image of a human face under a known directional illumination from a side, we represent the solution as a linear combination of basis shapes and recover the coefficients using a symmetry constraint on a facial shape and albedo. By solving a single least-squares system of equations, our algorithm provides a closed-form solution which satisfies both symmetry and statistical constraints in the best possible way. Our procedure takes only a few seconds, accounts for varying facial albedo, and is simpler than the previous methods. In the special case of horizontal illuminant direction, our algorithm runs even as fast as matrix-vector multiplication.
---
paper_title: A morphable model for the synthesis of 3D faces

paper_content:

In this paper, a new technique for modeling textured 3D faces is introduced. 3D faces can either be generated automatically from one or more photographs, or modeled directly through an intuitive user interface. Users are assisted in two key problems of computer aided face modeling. First, new face images or new 3D face models can be registered automatically by computing dense one-to-one correspondence to an internal face model. Second, the approach regulates the naturalness of modeled faces avoiding faces with an “unlikely” appearance. Starting from an example set of 3D face models, we derive a morphable face model by transforming the shape and texture of the examples into a vector space representation. New faces and expressions can be modeled by forming linear combinations of the prototypes. Shape and texture constraints derived from the statistics of our example faces are used to guide manual modeling or automated matching algorithms. We show 3D face reconstructions from single images and their applications for photo-realistic image manipulations. We also demonstrate face manipulations according to complex parameters such as gender, fullness of a face or its distinctiveness.
---
paper_title: A morphable model for the synthesis of 3D faces

paper_content:

In this paper, a new technique for modeling textured 3D faces is introduced. 3D faces can either be generated automatically from one or more photographs, or modeled directly through an intuitive user interface. Users are assisted in two key problems of computer aided face modeling. First, new face images or new 3D face models can be registered automatically by computing dense one-to-one correspondence to an internal face model. Second, the approach regulates the naturalness of modeled faces avoiding faces with an “unlikely” appearance. Starting from an example set of 3D face models, we derive a morphable face model by transforming the shape and texture of the examples into a vector space representation. New faces and expressions can be modeled by forming linear combinations of the prototypes. Shape and texture constraints derived from the statistics of our example faces are used to guide manual modeling or automated matching algorithms. We show 3D face reconstructions from single images and their applications for photo-realistic image manipulations. We also demonstrate face manipulations according to complex parameters such as gender, fullness of a face or its distinctiveness.
---
paper_title: Model-based 3D face capture with shape-from-silhouettes

paper_content:

We present a method for 3D face acquisition using a set or sequence of 2D binary silhouettes. Since silhouette images depend only on the shape and pose of an object, they are immune to lighting and/or texture variations (unlike feature or texture-based shape-from-correspondence). Our prior 3D face model is a linear combination of "eigenheads" obtained by applying PCA to a training set of laser-scanned 3D faces. These shape coefficients are the parameters for a near-automatic system for capturing the 3D shape as well as the 2D texture-map of a novel input face. Specifically, we use back-projection and a boundary-weighted XOR-based cost function for binary silhouette matching, coupled with a probabilistic "downhill-simplex" optimization for shape estimation and refinement. Experiments with a multicamera rig as well as monocular video sequences demonstrate the advantages of our 3D modeling framework and ultimately, its utility for robust face recognition with built-in invariance to pose and illumination.
---
paper_title: Interpreting face images using active appearance models

paper_content:

We demonstrate a fast, robust method of interpreting face images using an Active Appearance Model (AAM). An AAM contains a statistical model of shape and grey level appearance which can generalise to almost any face. Matching to an image involves finding model parameters which minimise the difference between the image and a synthesised face. We observe that displacing each model parameter from the correct value induces a particular pattern in the residuals. In a training phase, the AAM learns a linear model of the correlation between parameter displacements and the induced residuals. During search it measures the residuals and uses this model to correct the current parameters, leading to a better fit. A good overall match is obtained in a few iterations, even from poor starting estimates. We describe the technique in detail and show it matching to new face images.
---
paper_title: Active appearance models

paper_content:

We describe a new method of matching statistical models of appearance to images. A set of model parameters control modes of shape and gray-level variation learned from a training set. We construct an efficient iterative matching algorithm by learning the relationship between perturbations in the model parameters and the induced image errors.
---
paper_title: Face Reconstruction Across Different Poses and Arbitrary Illumination Conditions

paper_content:

In this paper, we present a novel method for face reconstruction from multi-posed face images taken under arbitrary unknown illumination conditions. Previous work shows that any face image can be represented by a set of low dimensional parameters: shape parameters, spherical harmonic basis (SHB) parameters, pose parameters and illumination coefficients. Thus, face reconstruction can be performed by recovering the set of parameters from the input images. In this paper, we demonstrate that the shape and SHB parameters can be estimated by minimizing the silhouettes errors and image intensity errors in a fast and robust manner. We propose a new algorithm to detect the corresponding points between the 3D face model and the input images by using silhouettes. We also apply a model-based bundle adjustment technique to perform this minimization. We provide a series of experiments on both synthetic and real data and experimental results show that our method can have an accurate face shape and texture reconstruction.
---
paper_title: Silhouette-Based 3D Face Shape Recovery

paper_content:

The creation of realistic 3D face models is still a fundamental problem in computer graphics. In this paper we present a novel method to obtain the 3D shape of an arbitrary human face using a sequence of silhouette images as input. Our face model is a linear combination of eigenheads, which are obtained by a Principal Component Analysis (PCA) of laser-scanned 3D human faces. The coefficients of this linear decomposition are used as our model parameters. We introduce a near-automatic method for reconstructing a 3D face model whose silhouette images match closest to the set of input silhouettes.
---
paper_title: Face image analysis using a multiple features fitting strategy

paper_content:

The main contribution of this thesis is a novel algorithm for ﬁtting a Three-Dimensional Morphable Model of faces to a 2D input image. This ﬁtting algorithm enables the estimation of the 3D shape, the texture, the 3D pose and the light direction from a single input image. Generally, the algorithms tackling the problem of 3D shape estimation from image data use only the pixels intensity as input to drive the estimation process. This was previously achieved using either a simple model, such as the Lambertian reﬂectance model, leading to a linear ﬁtting algorithm. Alternatively, this problem was addressed using a more precise model and minimizing a non-convex cost function with many local minima. One way to reduce the local minima problem is to use a stochastic optimization algorithm. However, the convergence properties (such as the radius of convergence) of such algorithms, are limited. Here, as well as the pixel intensity, we use various image features such as the edges or the location of the specular highlights. The 3D shape, texture and imaging parameters are then estimated by maximizing the posterior of the parameters given these image features. The overall cost function obtained is smoother and, hence, a stochastic optimization algorithm is not needed to avoid the local minima problem. This leads to the Multi-Features Fitting algorithm that has a wider radius of convergence and a higher level of precision. The new Multi-Feature Fitting algorithm is applied for such tasks as face identiﬁcation, facial expression transfer from one image to another image (of diﬀerent individuals) and face tracking across 3D pose and expression variations. The second contribution of this thesis is a careful comparison of well known ﬁtting algorithms used in the context of face modelling and recognition. It is shown that these algorithms achieve high run time eﬃciency at the cost of accuracy and generality (few face images may be analysed). The third and last contribution is the Matlab Morphable Model toolbox, a set of software tools developed in the Matlab programing environment. It allows (i) the generation of 3D faces from model parameters, (ii) the rendering of 3D faces, (iii) the ﬁtting of an input image using the Multi-Features Fitting algorithm and (iv) identiﬁcation from model parameters. The toolbox has a modular design that allows anyone to builds on it and, for instance, to improve the ﬁtting algorithm by incorporating new features in the cost function.
---
paper_title: Estimation of 3 D Faces and Illumination from Single Photographs Using a Bilinear Illumination Model

paper_content:

3D Face modeling is still one of the biggest challenges in computer graphics. In this paper we present a novel framework that acquires the 3D shape, texture, pose and illumination of a face from a single photograph. Additionally, we show how we can recreate a face under varying illumination conditions. Or, essentially relight it. Using a custom-built face scanning system, we have collected 3D face scans and light reflection images of a large and diverse group of human subjects. We derive a morphable face model for 3D face shapes and accompanying textures by transforming the data into a linear vector sub-space. The acquired images of faces under variable illumination are then used to derive a bilinear illumination model that spans 3D face shape and illumination variations. Using both models we, in turn, propose a novel fitting framework that estimates the parameters of the morphable model given a single photograph. Our framework can deal with complex face reflectance and lighting environments in an efficient and robust manner. In the results section of our paper, we compare our methods to existing ones and demonstrate its efficacy in reconstructing 3D face models when provided with a single photograph. We also provide several examples of facial relighting (on 2D images) by performing adequate decomposition of the estimated illumination using our framework.
---
paper_title: Estimating Coloured 3D Face Models from Single Images: An Example Based Approach

paper_content:

In this paper we present a method to derive 3D shape and surface texture of a human face from a single image. The method draws on a general flexible 3D face model which is “learned” from examples of individual 3D-face data (Cyberware-scans). In an analysis-by-synthesis loop, the flexible model is matched to the novel face image.
---
paper_title: Automatic 3D face modeling from video

paper_content:

In this paper, we develop an efficient technique for fully automatic recovery of accurate 3D face shape from videos captured by a low cost camera. The method is designed to work with a short video containing a face rotating from frontal view to profile view. The whole approach consists of three components. First, automatic initialization is performed in the first frame with approximately frontal face. Then, to handle the case of low quality image captured by low cost camera, the 2D feature matching, head poses and underlying 3D face shape are estimated and refined iteratively in an efficient way based on image sequence segmentation. Finally, to take advantage of the sparse structure of the proposed algorithm, sparse bundle adjustment technique is further employed to speed up the computation. We demonstrate the accuracy and robustness of the algorithm using a set of experiments
---
paper_title: Statistical symmetric shape from shading for 3d structure recovery of faces

paper_content:

In this paper, we aim to recover the 3D shape of a human face using a single image. We use a combination of symmetric shape from shading by Zhao and Chellappa and statistical approach for facial shape reconstruction by Atick, Griffin and Redlich. Given a single frontal image of a human face under a known directional illumination from a side, we represent the solution as a linear combination of basis shapes and recover the coefficients using a symmetry constraint on a facial shape and albedo. By solving a single least-squares system of equations, our algorithm provides a closed-form solution which satisfies both symmetry and statistical constraints in the best possible way. Our procedure takes only a few seconds, accounts for varying facial albedo, and is simpler than the previous methods. In the special case of horizontal illuminant direction, our algorithm runs even as fast as matrix-vector multiplication.
---
paper_title: A morphable model for the synthesis of 3D faces

paper_content:

In this paper, a new technique for modeling textured 3D faces is introduced. 3D faces can either be generated automatically from one or more photographs, or modeled directly through an intuitive user interface. Users are assisted in two key problems of computer aided face modeling. First, new face images or new 3D face models can be registered automatically by computing dense one-to-one correspondence to an internal face model. Second, the approach regulates the naturalness of modeled faces avoiding faces with an “unlikely” appearance. Starting from an example set of 3D face models, we derive a morphable face model by transforming the shape and texture of the examples into a vector space representation. New faces and expressions can be modeled by forming linear combinations of the prototypes. Shape and texture constraints derived from the statistics of our example faces are used to guide manual modeling or automated matching algorithms. We show 3D face reconstructions from single images and their applications for photo-realistic image manipulations. We also demonstrate face manipulations according to complex parameters such as gender, fullness of a face or its distinctiveness.
---
paper_title: Surfaces over Dirichlet tessellations

paper_content:

Abstract We develop a new class of surfaces, based on the concept of Bezier simplices expressed in terms of Sibson's natural neighbor coordinates. We also show how these surfaces may be utilized for scattered data interpolation.
---
paper_title: Automatic creation of 3D facial models

paper_content:

Model-based encoding of human facial features for narrowband visual communication is described. Based on an already prepared 3D human model, this coding method detects and understands a person's body motion and facial expressions. It expresses the essential information as compact codes and transmits it. At the receiving end, this code becomes the basis for modifying the 3D model of the person and thereby generating lifelike human images. The feature extraction used by the system to acquire data for regions or edges that express the eyes, nose, mouth, and outlines of the face and hair is discussed. The way in which the system creates a 3D model of the person by using the features extracted in the first part to modify a generic head model is also discussed. >
---
paper_title: Which stereo matching algorithm for accurate 3d face creation

paper_content:

This paper compares the efficiency of several stereo matching algorithms in reconstructing 3D faces from both real and synthetic stereo pairs. The stereo image acquisition system setup and the creation of a face disparity map benchmark image are detailed. Ground truth is build by visual matching of corresponding nodes of a dense colour grid projected onto the faces. This experiment was also performed on a human face model created using OpenGL with mapped texture to create as perfect as possible a set for evaluation, instead of real human faces like our previous experiments. Performance of the algorithms is measured by deviations of the reconstructed surfaces from a ground truth prototype. This experiment shows that contrary to expectations, there is seemingly very little difference between the currently most known stereo algorithms in the case of the human face reconstruction. It is shown that by combining the most efficient but slow graph-cut algorithm with fast dynamic programming, more accurate reconstruction results can be obtained.
---
paper_title: From 2D images to 3D face geometry

paper_content:

This paper presents a global scheme for 3D face reconstruction and face segmentation into a limited number of analytical patches from stereo images. From a depth map, we generate a 3D model of the face which is iteratively deformed under stereo and shape-from-shading constraints as well as differential features. This model enables us to improve the quality of the depth map, from which we perform the segmentation and the approximation of the surface.
---
paper_title: Regularized Bundle-Adjustment to Model Heads from Image Sequences without Calibration Data

paper_content:

We address the structure-from-motion problem in the context of head modeling from video sequences for which calibration data is not available. This task is made challenging by the fact that correspondences are difficult to establish due to lack of texture and that a quasi-euclidean representation is required for realism.We have developed an approach based on regularized bundle-adjustment. It takes advantage of our rough knowledge of the head's shape, in the form of a generic face model. It allows us to recover relative head-motion and epipolar geometry accurately and consistently enough to exploit a previously-developed stereo-based approach to head modeling. In this way, complete and realistic head models can be acquired with a cheap and entirely passive sensor, such as an ordinary video camera, with minimal manual intervention.We chose to demonstrate and evaluate our technique mainly in the context of head-modeling. We do so because it is the application for which all the tools required to perform the complete reconstruction are available to us. We will, however, argue that the approach is generic and could be applied to other tasks, such as body modeling, for which generic facetized models exist.
---
paper_title: Stereo matching with energy-minimizing snake grid for 3D face modeling

paper_content:

An energy minimizing snake algorithm that runs over a grid is designed and used to reconstruct high resolution 3D human faces from pairs of stereo images. The accuracy of reconstructed 3D data from stereo depends highly on how well stereo correspondences are established during the feature matching step. Establishing stereo correspondences on human faces is often ill posed and hard to achieve because of uniform texture, slow changes in depth, occlusion, and lack of gradient. We designed an energy minimizing algorithm that accurately finds correspondences on face images despite the aforementioned characteristics. The algorithm helps establish stereo correspondences unambiguously by applying a coarse-to-fine energy minimizing snake in grid format and yields a high resolution reconstruction at nearly every point of the image. Initially, the grid is stabilized using matches at a few selected high confidence edge points. The grid then gradually and consistently spreads over the low gradient regions of the image to reveal the accurate depths of object points. The grid applies its internal energy to approximate mismatches in occluded and noisy regions and to maintain smoothness of the reconstructed surfaces. The grid works in such a way that with every increment in reconstruction resolution, less time is required to establish correspondences. The snake used the curvature of the grid and gradient of image regions to automatically select its energy parameters and approximate the unmatched points using matched points from previous iterations, which also accelerates the overall matching process. The algorithm has been applied for the reconstruction of 3D human faces, and experimental results demonstrate the effectiveness and accuracy of the reconstruction.
---
paper_title: A Taxonomy and Evaluation of Dense Two-Frame Stereo Correspondence Algorithms

paper_content:

Stereo matching is one of the most active research areas in computer vision. While a large number of algorithms for stereo correspondence have been developed, relatively little work has been done on characterizing their performance. In this paper, we present a taxonomy of dense, two-frame stereo methods designed to assess the different components and design decisions made in individual stereo algorithms. Using this taxonomy, we compare existing stereo methods and present experiments evaluating the performance of many different variants. In order to establish a common software platform and a collection of data sets for easy evaluation, we have designed a stand-alone, flexible C++ implementation that enables the evaluation of individual components and that can be easily extended to include new algorithms. We have also produced several new multiframe stereo data sets with ground truth, and are making both the code and data sets available on the Web.
---
paper_title: Realistic video avatar

paper_content:

We present a system for the implementation of a photorealistic avatar using video captured from a user. This is achieved by constructing the dynamic video texture map and combining it with the 3D mesh model of the user to render the photorealistic avatar. The dynamic video texture map reflects the user's facial expressions and is generated by composing the cylindrical texture map of the user with constant updates either directly from the input video or derived from a temporal and spatial interpolation scheme. To derive the 3D mesh model for the user, we define feature points on a generic model and use the feature points identified on the video sequence to deform and obtain the avatar model. The goal of this realistic video avatar project is to provide a vivid representation of participants with a more realistic quality avatar, as compared to using only the facial animation parameters defined in MPEG-4 without the corresponding image updates. This scheme is suitable for conferencing applications because it requires much lower bandwidth than live video, and yet provides a 3D avatar representation for any virtual environment.
---
paper_title: 3D face reconstruction from video using a generic model

paper_content:

Reconstructing a 3D model of a human face from a video sequence is an important problem in computer vision, with applications to recognition, surveillance, multimedia etc. However, the quality of 3D reconstructions using structure from motion (SfM) algorithms is often not satisfactory. One common method of overcoming this problem is to use a generic model of a face. Existing work using this approach initializes the reconstruction algorithm with this generic model. The problem with this approach is that the algorithm can converge to a solution very close to this initial value, resulting in a reconstruction which resembles the generic model rather than the particular face in the video which needs to be modeled. We propose a method of 3D reconstruction of a human face from video in which the 3D reconstruction algorithm and the generic model are handled separately. A 3D estimate is obtained purely from the video sequence using SfM algorithms without use of the generic model. The final 3D model is obtained after combining the SfM estimate and the generic model using an energy function that corrects for the errors in the estimate by comparing local regions in the two models. The optimization is done using a Markov chain Monte Carlo (MCMC) sampling strategy. The main advantage of our algorithm over others is that it is able to retain the specific features of the face in the video sequence even when these features are different from those of the generic model. The evolution of the 3D model through the various stages of the algorithm is presented.
---
paper_title: Rapid modeling of animated faces from video

paper_content:

Generating realistic 3D human face models and facial animations has been a persistent challenge in computer graphics. We have developed a system that constructs textured 3Dface models from videos with minimal user interaction. Our system takes images andvideo sequences of a face with an ordinary video camera. After five manual clicks ontwo images to tell the system where the eye corners, nose top and mouth corners are, thesystem automatically generates a realistic looking 3D human head model and the constructed model can be animated immediately. A user with a PC and an ordinary camera can use our system to generate his/her face model in a few minutes. Copyright © 2001 John Wiley & Sons, Ltd.
---
paper_title: Recovering non-rigid 3D shape from image streams

paper_content:

The paper addresses the problem of recovering 3D non-rigid shape models from image sequences. For example, given a video recording of a talking person, we would like to estimate a 3D model of the lips and the full face and its internal modes of variation. Many solutions that recover 3D shape from 2D image sequences have been proposed; these so-called structure-from-motion techniques usually assume that the 3D object is rigid. For example, C. Tomasi and T. Kanades' (1992) factorization technique is based on a rigid shape matrix, which produces a tracking matrix of rank 3 under orthographic projection. We propose a novel technique based on a non-rigid model, where the 3D shape in each frame is a linear combination of a set of basis shapes. Under this model, the tracking matrix is of higher rank, and can be factored in a three-step process to yield pose, configuration and shape. To the best of our knowledge, this is the first model free approach that can recover from single-view video sequences nonrigid shape models. We demonstrate this new algorithm on several video sequences. We were able to recover 3D non-rigid human face and animal models with high accuracy.
---
paper_title: Automatic 3D face modeling from video

paper_content:

In this paper, we develop an efficient technique for fully automatic recovery of accurate 3D face shape from videos captured by a low cost camera. The method is designed to work with a short video containing a face rotating from frontal view to profile view. The whole approach consists of three components. First, automatic initialization is performed in the first frame with approximately frontal face. Then, to handle the case of low quality image captured by low cost camera, the 2D feature matching, head poses and underlying 3D face shape are estimated and refined iteratively in an efficient way based on image sequence segmentation. Finally, to take advantage of the sparse structure of the proposed algorithm, sparse bundle adjustment technique is further employed to speed up the computation. We demonstrate the accuracy and robustness of the algorithm using a set of experiments
---
paper_title: Extracting Structure from Optical Flow Using the Fast Error Search Technique

paper_content:

AbstractIn this paper, we present a globally optimal and computationally efficient technique for estimating the focus of expansion (FOE) of an optical flow field, using fast partial search. For each candidate location on a discrete sampling of the image area, we generate a linear system of equations for determining the remaining unknowns, viz. rotation and inverse depth. We compute the least squares error of the system without actually solving the equations, to generate an error surface that describes the goodness of fit across the hypotheses. Using Fourier techniques, we prove that given an N × N flow field, the FOE, and subsequently rotation and structure, can be estimated in ::: $$\mathcal{O}(N^2 \log N)$$ ::: operations. Since the resulting system is linear, bounded perturbations in the data lead to bounded errors.We support the theoretical development and proof of our technique with experiments on synthetic and real data. Through a series of experiments on synthetic data, we prove the correctness, robustness and operating envelope of our algorithm. We demonstrate the utility of our technique by applying it for detecting obstacles from a monocular sequence of images.
---
paper_title: Object tracking: A survey

paper_content:

The goal of this article is to review the state-of-the-art tracking methods, classify them into different categories, and identify new trends. Object tracking, in general, is a challenging problem. Difficulties in tracking objects can arise due to abrupt object motion, changing appearance patterns of both the object and the scene, nonrigid object structures, object-to-object and object-to-scene occlusions, and camera motion. Tracking is usually performed in the context of higher-level applications that require the location and/or shape of the object in every frame. Typically, assumptions are made to constrain the tracking problem in the context of a particular application. In this survey, we categorize the tracking methods on the basis of the object and motion representations used, provide detailed descriptions of representative methods in each category, and examine their pros and cons. Moreover, we discuss the important issues related to tracking including the use of appropriate image features, selection of motion models, and detection of objects.
---
paper_title: A morphable model for the synthesis of 3D faces

paper_content:

In this paper, a new technique for modeling textured 3D faces is introduced. 3D faces can either be generated automatically from one or more photographs, or modeled directly through an intuitive user interface. Users are assisted in two key problems of computer aided face modeling. First, new face images or new 3D face models can be registered automatically by computing dense one-to-one correspondence to an internal face model. Second, the approach regulates the naturalness of modeled faces avoiding faces with an “unlikely” appearance. Starting from an example set of 3D face models, we derive a morphable face model by transforming the shape and texture of the examples into a vector space representation. New faces and expressions can be modeled by forming linear combinations of the prototypes. Shape and texture constraints derived from the statistics of our example faces are used to guide manual modeling or automated matching algorithms. We show 3D face reconstructions from single images and their applications for photo-realistic image manipulations. We also demonstrate face manipulations according to complex parameters such as gender, fullness of a face or its distinctiveness.
---
paper_title: From Regular Images to Animated Heads: A Least Squares Approach

paper_content:

We show that we can effectively fit arbitrarily complex animation models to noisy image data. Our approach is based on leastsquares adjustment using of a set of progressively finer control triangulations and takes advantage of three complementary sources of information: stereo data, silhouette edges and 2-D feature points.
---
paper_title: Model-based 3D face capture with shape-from-silhouettes

paper_content:

We present a method for 3D face acquisition using a set or sequence of 2D binary silhouettes. Since silhouette images depend only on the shape and pose of an object, they are immune to lighting and/or texture variations (unlike feature or texture-based shape-from-correspondence). Our prior 3D face model is a linear combination of "eigenheads" obtained by applying PCA to a training set of laser-scanned 3D faces. These shape coefficients are the parameters for a near-automatic system for capturing the 3D shape as well as the 2D texture-map of a novel input face. Specifically, we use back-projection and a boundary-weighted XOR-based cost function for binary silhouette matching, coupled with a probabilistic "downhill-simplex" optimization for shape estimation and refinement. Experiments with a multicamera rig as well as monocular video sequences demonstrate the advantages of our 3D modeling framework and ultimately, its utility for robust face recognition with built-in invariance to pose and illumination.
---
paper_title: Face Reconstruction Across Different Poses and Arbitrary Illumination Conditions

paper_content:

In this paper, we present a novel method for face reconstruction from multi-posed face images taken under arbitrary unknown illumination conditions. Previous work shows that any face image can be represented by a set of low dimensional parameters: shape parameters, spherical harmonic basis (SHB) parameters, pose parameters and illumination coefficients. Thus, face reconstruction can be performed by recovering the set of parameters from the input images. In this paper, we demonstrate that the shape and SHB parameters can be estimated by minimizing the silhouettes errors and image intensity errors in a fast and robust manner. We propose a new algorithm to detect the corresponding points between the 3D face model and the input images by using silhouettes. We also apply a model-based bundle adjustment technique to perform this minimization. We provide a series of experiments on both synthetic and real data and experimental results show that our method can have an accurate face shape and texture reconstruction.
---
paper_title: Silhouette-Based 3D Face Shape Recovery

paper_content:

The creation of realistic 3D face models is still a fundamental problem in computer graphics. In this paper we present a novel method to obtain the 3D shape of an arbitrary human face using a sequence of silhouette images as input. Our face model is a linear combination of eigenheads, which are obtained by a Principal Component Analysis (PCA) of laser-scanned 3D human faces. The coefficients of this linear decomposition are used as our model parameters. We introduce a near-automatic method for reconstructing a 3D face model whose silhouette images match closest to the set of input silhouettes.
---
paper_title: Animated Heads from Ordinary Images: A Least Squares Approach

paper_content:

We show that we can effectively fit arbitrarily complex animation models to noisy data extracted from ordinary face images. Our approach is based on least-squares adjustment, using of a set of progressively finer control triangulations and takes advantage of three complementary sources of information: stereo data, silhouette edges, and 2D feature points. In this way, complete head models?including ears and hair?can be acquired with a cheap and entirely passive sensor, such as an ordinary video camera. They can then be fed to existing animation software to produce synthetic sequences.
---
paper_title: A morphable model for the synthesis of 3D faces

paper_content:

In this paper, a new technique for modeling textured 3D faces is introduced. 3D faces can either be generated automatically from one or more photographs, or modeled directly through an intuitive user interface. Users are assisted in two key problems of computer aided face modeling. First, new face images or new 3D face models can be registered automatically by computing dense one-to-one correspondence to an internal face model. Second, the approach regulates the naturalness of modeled faces avoiding faces with an “unlikely” appearance. Starting from an example set of 3D face models, we derive a morphable face model by transforming the shape and texture of the examples into a vector space representation. New faces and expressions can be modeled by forming linear combinations of the prototypes. Shape and texture constraints derived from the statistics of our example faces are used to guide manual modeling or automated matching algorithms. We show 3D face reconstructions from single images and their applications for photo-realistic image manipulations. We also demonstrate face manipulations according to complex parameters such as gender, fullness of a face or its distinctiveness.
---
paper_title: Automatic 3D reconstruction for face recognition

paper_content:

An analysis-by-synthesis framework for face recognition with variant pose, illumination and expression (PIE) is proposed in this paper. First, an efficient 2D-to-3D integrated face reconstruction approach is introduced to reconstruct a personalized 3D face model from a single frontal face image with neutral expression and normal illumination. Then, realistic virtual faces with different PIE are synthesized based on the personalized 3D face to characterize the face subspace. Finally, face recognition is conducted based on these representative virtual faces. Compared with other related works, this framework has the following advantages: 1) only one single frontal face is required for face recognition, which avoids the burdensome enrollment work; 2) the synthesized face samples provide the capability to conduct recognition under difficult conditions like complex PIE; and 3) the proposed 2D-to-3D integrated face reconstruction approach is fully automatic and more efficient. The extensive experimental results show that the synthesized virtual faces significantly improve the accuracy of face recognition with variant PIE.
---
paper_title: Illumination-insensitive face recognition using symmetric shape-from-shading

paper_content:

Sensitivity to variations in illumination is a fundamental and challenging problem in face recognition. In this paper, we describe a new method based on symmetric shape-from-shading (SSFS) to develop a face recognition system that is robust to changes in illumination. The basic idea of this approach is to use the SSFS algorithm as a tool to obtain a prototype image which is illumination-normalized. It has been shown that the SSFS algorithm has a unique point-wise solution. But it is still difficult to recover accurate shape information given a single real face image with complex shape and varying albedo. In stead, we utilize the fact that all faces share a similar shape making the direct computation of the prototype image from a given face image feasible. Finally, to demonstrate the efficacy of our method, we have applied it to several publicly available face databases.
---
paper_title: Efficient 3D reconstruction for face recognition

paper_content:

Face recognition with variant pose, illumination and expression (PIE) is a challenging problem. In this paper, we propose an analysis-by-synthesis framework for face recognition with variant PIE. First, an efficient two-dimensional (2D)-to-three-dimensional (3D) integrated face reconstruction approach is introduced to reconstruct a personalized 3D face model from a single frontal face image with neutral expression and normal illumination. Then, realistic virtual faces with different PIE are synthesized based on the personalized 3D face to characterize the face subspace. Finally, face recognition is conducted based on these representative virtual faces. Compared with other related work, this framework has following advantages: (1) only one single frontal face is required for face recognition, which avoids the burdensome enrollment work; (2) the synthesized face samples provide the capability to conduct recognition under difficult conditions like complex PIE; and (3) compared with other 3D reconstruction approaches, our proposed 2D-to-3D integrated face reconstruction approach is fully automatic and more efficient. The extensive experimental results show that the synthesized virtual faces significantly improve the accuracy of face recognition with changing PIE.
---
paper_title: Face recognition from 2D and 3D images using 3D Gabor filters

paper_content:

To recognize faces with different facial expressions or varying views from only one stored prototype per person is challenging. This paper presents such a system based on both 3D range data as well as the corresponding 2D gray-level facial images. The traditional 3D Gabor filter (3D TGF) is explored in the face recognition domain to extract expression-invariant features. To extract view-invariant features, a rotation-invariant 3D spherical Gabor filter (3D SGF) is proposed. Furthermore, a two-dimensional (2D) Gabor histogram is employed to represent the Gabor responses of the 3D SGF for solving the missing-point problem caused by self-occlusions under large rotation angles. The choice of 3D Gabor filter parameters for face recognition is examined as well. To match a given test face with each model face, the Least Trimmed Square Hausdorff Distance (LTS-HD) is employed to tackle the possible partial-matching problem. Experimental results based on our face database involving 80 persons have demonstrated that our approach outperforms the standard Eigenface approach and the approach using the 2D Gabor-wavelets representation.
---
paper_title: Silhouette-Based 3D Face Shape Recovery

paper_content:

The creation of realistic 3D face models is still a fundamental problem in computer graphics. In this paper we present a novel method to obtain the 3D shape of an arbitrary human face using a sequence of silhouette images as input. Our face model is a linear combination of eigenheads, which are obtained by a Principal Component Analysis (PCA) of laser-scanned 3D human faces. The coefficients of this linear decomposition are used as our model parameters. We introduce a near-automatic method for reconstructing a 3D face model whose silhouette images match closest to the set of input silhouettes.
---
paper_title: Prototyping and Transforming Facial Textures for Perception Research

paper_content:

Transforming facial images along perceived dimensions (such as age, gender, race, or health) has application in areas as diverse as psychology, medicine, and forensics. We can use prototype images to define the salient features of a particular face classification (for example, European female adult or East-Asian male child). We then use the differences between two prototypes to define an axis of transformation, such as younger to older. By applying these changes to a given input face, we can change its apparent age, race, or gender. Psychological investigations reveal a limitation with existing methods that's particularly apparent when changing the age of faces. We relate the problem to the loss of facial textures (such as stubble and wrinkles) in the prototypes due to the blending process. We review the existing face prototyping and transformation methods and present a new, wavelet-based method for prototyping and transforming facial textures.
---

Title: IMAGE BASED 3D FACE RECONSTRUCTION: A SURVEY

Section 1: Introduction
Description 1: Provide an overview of the significance of face image processing and the motivation behind transitioning from 2D to 3D face reconstruction.

Section 2: Terminology
Description 2: Define essential terminology and concepts related to 3D face reconstruction, including parametric surfaces, triangle meshes, range images, and texture mapping methods.

Section 3: General 3D Face Reconstruction Topics
Description 3: Discuss general aspects encountered in different 3D face reconstruction algorithms that are frequently referenced in literature.

Section 4: 3D Face Databases
Description 4: Highlight the importance, usage, and requirements of 3D face databases in 3D face reconstruction methods.

Section 5: Establishing 3D Face Correspondences
Description 5: Review techniques used for establishing vertex correspondences among 3D faces in a database to achieve alignment of training samples.

Section 6: Feature Detection
Description 6: Describe the process and importance of locating key facial features in 2D images for use in 3D reconstruction algorithms.

Section 7: Assumptions
Description 7: Outline common assumptions made during 3D face reconstruction, such as types of projection, albedo, and light sources.

Section 8: Deformable 3D Models
Description 8: Cover different types of deformable models used in 3D face reconstruction, including wire-frame and statistical models.

Section 9: 3D Face Reconstruction Methodologies
Description 9: Categorize and detail various 3D face reconstruction methods, such as example-based, stereo, video, and silhouette-based methods.

Section 10: Performance Evaluation
Description 10: Examine methodologies and results for evaluating the performance and accuracy of 3D face reconstruction techniques.

Section 11: Discussion
Description 11: Discuss the strengths, limitations, and application potential of different 3D face reconstruction techniques, and their compliance with key criteria for real-life applications.

Section 12: Future Directions
Description 12: Identify the main research areas that need to be addressed to advance the field of image-based 3D face reconstruction.

Section 13: Conclusion
Description 13: Summarize the survey’s findings and the remaining challenges in the development of image-based 3D face reconstruction techniques.

Section 14: Acknowledgments
Description 14: Acknowledge the contributions and support received for the work presented in the survey.