Recent Advances on Singlemodal and Multimodal Face Recognition: A Survey
---
paper_title: 3D Face Recognition by Local Shape Difference Boosting

paper_content:

A new approach, called Collective Shape Difference Classifier(CSDC), is proposed to improve the accuracy and computational efficiency of 3D face recognition. The CSDC learns the most discriminative local areas from the Pure Shape Difference Map(PSDM) and trains them as weak classifiers for assembling a collective strong classifier using the real-boosting approach. The PSDM is established between two 3D face models aligned by a posture normalization procedure based on facial features. The model alignment is self-dependent, which avoids registering the probe face against every different gallery face during the recognition, so that a high computational speed is obtained. The experiments, carried out on the FRGC v2 and BU-3DFE databases, yield rank-1 recognition rates better than 98%. Each recognition against a gallery with 1000 faces only needs about 3.05seconds. These two experimental results together with the high performance recognition on partial faces demonstrate that our algorithm is not only effective but also efficient.
---
paper_title: 2D and 3D face recognition: A survey

paper_content:

Government agencies are investing a considerable amount of resources into improving security systems as result of recent terrorist events that dangerously exposed flaws and weaknesses in today's safety mechanisms. Badge or password-based authentication procedures are too easy to hack. Biometrics represents a valid alternative but they suffer of drawbacks as well. Iris scanning, for example, is very reliable but too intrusive; fingerprints are socially accepted, but not applicable to non-consentient people. On the other hand, face recognition represents a good compromise between what's socially acceptable and what's reliable, even when operating under controlled conditions. In last decade, many algorithms based on linear/nonlinear methods, neural networks, wavelets, etc. have been proposed. Nevertheless, Face Recognition Vendor Test 2002 shown that most of these approaches encountered problems in outdoor conditions. This lowered their reliability compared to state of the art biometrics. This paper provides an ''ex cursus'' of recent face recognition research trends in 2D imagery and 3D model based algorithms. To simplify comparisons across different approaches, tables containing different collection of parameters (such as input size, recognition rate, number of addressed problems) are provided. This paper concludes by proposing possible future directions.
---
paper_title: Three-Dimensional Face Recognition in the Presence of Facial Expressions: An Annotated Deformable Model Approach

paper_content:

In this paper, we present the computational tools and a hardware prototype for 3D face recognition. Full automation is provided through the use of advanced multistage alignment algorithms, resilience to facial expressions by employing a deformable model framework, and invariance to 3D capture devices through suitable preprocessing steps. In addition, scalability in both time and space is achieved by converting 3D facial scans into compact metadata. We present our results on the largest known, and now publicly available, face recognition grand challenge 3D facial database consisting of several thousand scans. To the best of our knowledge, this is the highest performance reported on the FRGC v2 database for the 3D modality
---
paper_title: A survey of approaches and challenges in 3D and multi-modal 3D + 2D face recognition

paper_content:

This survey focuses on recognition performed by matching models of the three-dimensional shape of the face, either alone or in combination with matching corresponding two-dimensional intensity images. Research trends to date are summarized, and challenges confronting the development of more accurate three-dimensional face recognition are identified. These challenges include the need for better sensors, improved recognition algorithms, and more rigorous experimental methodology.
---
paper_title: Facial Recognition Technology: A Survey of Policy and Implementation Issues

paper_content:

Facial recognition technology (FRT) has emerged as an attractive solution to address many contemporary needs for identification and the verification of identity claims. It brings together the promise of other biometric systems, which attempt to tie identity to individually distinctive features of the body, and the more familiar functionality of visual surveillance systems. This report develops a socio-political analysis that bridges the technical and social-scientific literatures on FRT and addresses the unique challenges and concerns that attend its development, evaluation, and specific operational uses, contexts, and goals. It highlights the potential and limitations of the technology, noting those tasks for which it seems ready for deployment, those areas where performance obstacles may be overcome by future technological developments or sound operating procedures, and still other issues which appear intractable. Its concern with efficacy extends to ethical considerations. For the purposes of this summary, the main findings and recommendations of the report are broken down into five broad categories: performance, evaluation, operation, policy concerns, and moral and political considerations. These findings and recommendations employ certain technical concepts and language that are explained and explored in the body of the report and glossary, to which you should turn for further elaboration.
---
paper_title: Thermal face recognition in an operational scenario

paper_content:

We present results on the latest advances in thermal infrared face recognition, and its use in combination with visible imagery. Previous research by the authors has shown high performance under very controlled conditions, or questionable performance under a wider range of conditions. This paper shows results on the use of thermal infrared and visible imagery for face recognition in operational scenarios. In particular, we show performance statistics for outdoor face recognition and recognition across multiple sessions. Our results support the conclusion that face recognition performance with thermal infrared imagery is stable over multiple sessions, and that fusion of modalities increases performance. As measured by the number of images and number of subjects, this is the largest ever reported study on thermal face recognition.
---
paper_title: 2D and 3D face recognition: A survey

paper_content:

Government agencies are investing a considerable amount of resources into improving security systems as result of recent terrorist events that dangerously exposed flaws and weaknesses in today's safety mechanisms. Badge or password-based authentication procedures are too easy to hack. Biometrics represents a valid alternative but they suffer of drawbacks as well. Iris scanning, for example, is very reliable but too intrusive; fingerprints are socially accepted, but not applicable to non-consentient people. On the other hand, face recognition represents a good compromise between what's socially acceptable and what's reliable, even when operating under controlled conditions. In last decade, many algorithms based on linear/nonlinear methods, neural networks, wavelets, etc. have been proposed. Nevertheless, Face Recognition Vendor Test 2002 shown that most of these approaches encountered problems in outdoor conditions. This lowered their reliability compared to state of the art biometrics. This paper provides an ''ex cursus'' of recent face recognition research trends in 2D imagery and 3D model based algorithms. To simplify comparisons across different approaches, tables containing different collection of parameters (such as input size, recognition rate, number of addressed problems) are provided. This paper concludes by proposing possible future directions.
---
paper_title: Face recognition with visible and thermal infrared imagery

paper_content:

We present a comprehensive performance study of multiple appearance-based face recognition methodologies, on visible and thermal infrared imagery. We compare algorithms within the same imaging modality as well as between them. Both identification and verification scenarios are considered, and appropriate performance statistics reported for each case. Our experimental design is aimed at gaining full understanding of algorithm performance under varying conditions, and is based on Monte Carlo analysis of performance measures. This analysis reveals that under many circumstances, using thermal infrared imagery yields higher performance, while in other cases performance in both modalities is equivalent. Performance increases further when algorithms on visible and thermal infrared imagery are fused. Our study also provides a partial explanation for the multiple contradictory claims in the literature regarding performance of various algorithms on visible data sets.
---
paper_title: A Survey of Face Recognition Techniques

paper_content:

Face recognition presents a challenging problem in the field of image analysis and computer vision, and as such has received a great deal of attention over the last few years because of its many applications in various domains. Face recognition techniques can be broadly divided into three categories based on the face data acquisition methodology: methods that operate on intensity images; those that deal with video sequences; and those that require other sensory data such as 3D information or infra-red imagery. In this paper, an overview of some of the well-known methods in each of these categories is provided and some of the benefits and drawbacks of the schemes mentioned therein are examined. Furthermore, a discussion outlining the incentive for using face recognition, the applications of this technology, and some of the difficulties plaguing current systems with regard to this task has also been provided. This paper also mentions some of the most recent algorithms developed for this purpose and attempts to give an idea of the state of the art of face recognition technology.
---
paper_title: 2D and 3D face recognition: A survey

paper_content:

Government agencies are investing a considerable amount of resources into improving security systems as result of recent terrorist events that dangerously exposed flaws and weaknesses in today's safety mechanisms. Badge or password-based authentication procedures are too easy to hack. Biometrics represents a valid alternative but they suffer of drawbacks as well. Iris scanning, for example, is very reliable but too intrusive; fingerprints are socially accepted, but not applicable to non-consentient people. On the other hand, face recognition represents a good compromise between what's socially acceptable and what's reliable, even when operating under controlled conditions. In last decade, many algorithms based on linear/nonlinear methods, neural networks, wavelets, etc. have been proposed. Nevertheless, Face Recognition Vendor Test 2002 shown that most of these approaches encountered problems in outdoor conditions. This lowered their reliability compared to state of the art biometrics. This paper provides an ''ex cursus'' of recent face recognition research trends in 2D imagery and 3D model based algorithms. To simplify comparisons across different approaches, tables containing different collection of parameters (such as input size, recognition rate, number of addressed problems) are provided. This paper concludes by proposing possible future directions.
---
paper_title: Discriminative common vectors for face recognition

paper_content:

In face recognition tasks, the dimension of the sample space is typically larger than the number of the samples in the training set. As a consequence, the within-class scatter matrix is singular and the linear discriminant analysis (LDA) method cannot be applied directly. This problem is known as the "small sample size" problem. In this paper, we propose a new face recognition method called the discriminative common vector method based on a variation of Fisher's linear discriminant analysis for the small sample size case. Two different algorithms are given to extract the discriminative common vectors representing each person in the training set of the face database. One algorithm uses the within-class scatter matrix of the samples in the training set while the other uses the subspace methods and the Gram-Schmidt orthogonalization procedure to obtain the discriminative common vectors. Then, the discriminative common vectors are used for classification of new faces. The proposed method yields an optimal solution for maximizing the modified Fisher's linear discriminant criterion given in the paper. Our test results show that the discriminative common vector method is superior to other methods in terms of recognition accuracy, efficiency, and numerical stability.
---
paper_title: Facial Recognition Technology: A Survey of Policy and Implementation Issues

paper_content:

Facial recognition technology (FRT) has emerged as an attractive solution to address many contemporary needs for identification and the verification of identity claims. It brings together the promise of other biometric systems, which attempt to tie identity to individually distinctive features of the body, and the more familiar functionality of visual surveillance systems. This report develops a socio-political analysis that bridges the technical and social-scientific literatures on FRT and addresses the unique challenges and concerns that attend its development, evaluation, and specific operational uses, contexts, and goals. It highlights the potential and limitations of the technology, noting those tasks for which it seems ready for deployment, those areas where performance obstacles may be overcome by future technological developments or sound operating procedures, and still other issues which appear intractable. Its concern with efficacy extends to ethical considerations. For the purposes of this summary, the main findings and recommendations of the report are broken down into five broad categories: performance, evaluation, operation, policy concerns, and moral and political considerations. These findings and recommendations employ certain technical concepts and language that are explained and explored in the body of the report and glossary, to which you should turn for further elaboration.
---
paper_title: A method for registration of 3-D shapes

paper_content:

The authors describe a general-purpose, representation-independent method for the accurate and computationally efficient registration of 3-D shapes including free-form curves and surfaces. The method handles the full six degrees of freedom and is based on the iterative closest point (ICP) algorithm, which requires only a procedure to find the closest point on a geometric entity to a given point. The ICP algorithm always converges monotonically to the nearest local minimum of a mean-square distance metric, and the rate of convergence is rapid during the first few iterations. Therefore, given an adequate set of initial rotations and translations for a particular class of objects with a certain level of 'shape complexity', one can globally minimize the mean-square distance metric over all six degrees of freedom by testing each initial registration. One important application of this method is to register sensed data from unfixtured rigid objects with an ideal geometric model, prior to shape inspection. Experimental results show the capabilities of the registration algorithm on point sets, curves, and surfaces. >
---
paper_title: 3D Face Recognition by Local Shape Difference Boosting

paper_content:

A new approach, called Collective Shape Difference Classifier(CSDC), is proposed to improve the accuracy and computational efficiency of 3D face recognition. The CSDC learns the most discriminative local areas from the Pure Shape Difference Map(PSDM) and trains them as weak classifiers for assembling a collective strong classifier using the real-boosting approach. The PSDM is established between two 3D face models aligned by a posture normalization procedure based on facial features. The model alignment is self-dependent, which avoids registering the probe face against every different gallery face during the recognition, so that a high computational speed is obtained. The experiments, carried out on the FRGC v2 and BU-3DFE databases, yield rank-1 recognition rates better than 98%. Each recognition against a gallery with 1000 faces only needs about 3.05seconds. These two experimental results together with the high performance recognition on partial faces demonstrate that our algorithm is not only effective but also efficient.
---
paper_title: Automatic 3D reconstruction for face recognition

paper_content:

An analysis-by-synthesis framework for face recognition with variant pose, illumination and expression (PIE) is proposed in this paper. First, an efficient 2D-to-3D integrated face reconstruction approach is introduced to reconstruct a personalized 3D face model from a single frontal face image with neutral expression and normal illumination. Then, realistic virtual faces with different PIE are synthesized based on the personalized 3D face to characterize the face subspace. Finally, face recognition is conducted based on these representative virtual faces. Compared with other related works, this framework has the following advantages: 1) only one single frontal face is required for face recognition, which avoids the burdensome enrollment work; 2) the synthesized face samples provide the capability to conduct recognition under difficult conditions like complex PIE; and 3) the proposed 2D-to-3D integrated face reconstruction approach is fully automatic and more efficient. The extensive experimental results show that the synthesized virtual faces significantly improve the accuracy of face recognition with variant PIE.
---
paper_title: Iterative Closest Normal Point for 3D Face Recognition

paper_content:

The common approach for 3D face recognition is to register a probe face to each of the gallery faces and then calculate the sum of the distances between their points. This approach is computationally expensive and sensitive to facial expression variation. In this paper, we introduce the iterative closest normal point method for finding the corresponding points between a generic reference face and every input face. The proposed correspondence finding method samples a set of points for each face, denoted as the closest normal points. These points are effectively aligned across all faces, enabling effective application of discriminant analysis methods for 3D face recognition. As a result, the expression variation problem is addressed by minimizing the within-class variability of the face samples while maximizing the between-class variability. As an important conclusion, we show that the surface normal vectors of the face at the sampled points contain more discriminatory information than the coordinates of the points. We have performed comprehensive experiments on the Face Recognition Grand Challenge database, which is presently the largest available 3D face database. We have achieved verification rates of 99.6 and 99.2 percent at a false acceptance rate of 0.1 percent for the all versus all and ROC III experiments, respectively, which, to the best of our knowledge, have seven and four times less error rates, respectively, compared to the best existing methods on this database.
---
paper_title: Expression-Invariant Representations of Faces

paper_content:

Addressed here is the problem of constructing and analyzing expression-invariant representations of human faces. We demonstrate and justify experimentally a simple geometric model that allows to describe facial expressions as isometric deformations of the facial surface. The main step in the construction of expression-invariant representation of a face involves embedding of the facial intrinsic geometric structure into some low-dimensional space. We study the influence of the embedding space geometry and dimensionality choice on the representation accuracy and argue that compared to its Euclidean counterpart, spherical embedding leads to notably smaller metric distortions. We experimentally support our claim showing that a smaller embedding error leads to better recognition.
---
paper_title: Face recognition based on frontal views generated from non-frontal images

paper_content:

This paper presents a method for face recognition across large changes in viewpoint. Our method is based on a morphable model of 3D faces that represents face-specific information extracted from a dataset of 3D scans. For non-frontal face recognition in 2D still images, the morphable model can be incorporated in two different approaches: in the first, it serves as a preprocessing step by estimating the 3D shape of novel faces from the non-frontal input images, and generating frontal views of the reconstructed faces at a standard illumination using 3D computer graphics. The transformed images are then fed into state-of-the-art face recognition systems that are optimized for frontal views. This method was shown to be extremely effective in the Face Recognition Vendor Test FRVT 2002. In the process of estimating the 3D shape of a face from an image, a set of model coefficients are estimated. In the second method, face recognition is performed directly from these coefficients. In this paper we explain the algorithm used to preprocess the images in FRVT 2002, present additional FRVT 2002 results, and compare these results to recognition from the model coefficients.
---
paper_title: Overview of the face recognition grand challenge

paper_content:

Over the last couple of years, face recognition researchers have been developing new techniques. These developments are being fueled by advances in computer vision techniques, computer design, sensor design, and interest in fielding face recognition systems. Such advances hold the promise of reducing the error rate in face recognition systems by an order of magnitude over Face Recognition Vendor Test (FRVT) 2002 results. The face recognition grand challenge (FRGC) is designed to achieve this performance goal by presenting to researchers a six-experiment challenge problem along with data corpus of 50,000 images. The data consists of 3D scans and high resolution still imagery taken under controlled and uncontrolled conditions. This paper describes the challenge problem, data corpus, and presents baseline performance and preliminary results on natural statistics of facial imagery.
---
paper_title: Point Signatures: A New Representation for 3D Object Recognition

paper_content:

Few systems capable of recognizing complex objects with free-form (sculptured) surfaces have been developed. The apparent lack of success is mainly due to the lack of a competent modelling scheme for representing such complex objects. In this paper, a new form of point representation for describing 3D free-form surfaces is proposed. This representation, which we call the point signature, serves to describe the structural neighbourhood of a point in a more complete manner than just using the 3D coordinates of the point. Being invariant to rotation and translation, the point signature can be used directly to hypothesize the correspondence to model points with similar signatures. Recognition is achieved by matching the signatures of data points representing the sensed surface to the signatures of data points representing the model surface. ::: ::: The use of point signatures is not restricted to the recognition of a single-object scene to a small library of models. Instead, it can be extended naturally to the recognition of scenes containing multiple partially-overlapping objects (which may also be juxtaposed with each other) against a large model library. No preliminary phase of segmenting the scene into the component objects is required. In searching for the appropriate candidate model, recognition need not proceed in a linear order which can become prohibitive for a large model library. For a given scene, signatures are extracted at arbitrarily spaced seed points. Each of these signatures is used to vote for models that contain points having similar signatures. Inappropriate models with low votes can be rejected while the remaining candidate models are ordered according to the votes they received. In this way, efficient verification of the hypothesized candidates can proceed by testing the most likely model first. Experiments using real data obtained from a range finder have shown fast recognition from a library of fifteen models whose complexities vary from that of simple piecewise quadric shapes to complicated face masks. Results from the recognition of both single-object and multiple-object scenes are presented.
---
paper_title: Multiple Nose Region Matching for 3D Face Recognition under Varying Facial Expression

paper_content:

An algorithm is proposed for 3D face recognition in the presence of varied facial expressions. It is based on combining the match scores from matching multiple overlapping regions around the nose. Experimental results are presented using the largest database employed to date in 3D face recognition studies, over 4,000 scans of 449 subjects. Results show substantial improvement over matching the shape of a single larger frontal face region. This is the first approach to use multiple overlapping regions around the nose to handle the problem of expression variation
---
paper_title: 3D Face Recognition Using Isogeodesic Stripes

paper_content:

In this paper, we present a novel approach to 3D face matching that shows high effectiveness in distinguishing facial differences between distinct individuals from differences induced by nonneutral expressions within the same individual. The approach takes into account geometrical information of the 3D face and encodes the relevant information into a compact representation in the form of a graph. Nodes of the graph represent equal width isogeodesic facial stripes. Arcs between pairs of nodes are labeled with descriptors, referred to as 3D Weighted Walkthroughs (3DWWs), that capture the mutual relative spatial displacement between all the pairs of points of the corresponding stripes. Face partitioning into isogeodesic stripes and 3DWWs together provide an approximate representation of local morphology of faces that exhibits smooth variations for changes induced by facial expressions. The graph-based representation permits very efficient matching for face recognition and is also suited to being employed for face identification in very large data sets with the support of appropriate index structures. The method obtained the best ranking at the SHREC 2008 contest for 3D face recognition. We present an extensive comparative evaluation of the performance with the FRGC v2.0 data set and the SHREC08 data set.
---
paper_title: Face Recognition with 3D Model-Based Synthesis

paper_content:

Current appearance-based face recognition system encounters the difficulty to recognize faces with appearance variations, while only a small number of training images are available. We present a scheme based on the analysis by synthesis framework. A 3D generic face model is aligned onto a given frontal face image. A number of synthetic face images are generated with appearance variations from the aligned 3D face model. These synthesized images are used to construct an affine subspace for each subject. Training and test images for each subject are represented in the same way in such a subspace. Face recognition is achieved by minimizing the distance between the subspace of a test subject and that of each subject in the database. Only a single face image of each subject is available for training in our experiments. Preliminary experimental results are promising.
---
paper_title: 3D Face Recognition Using 3D Alignment for PCA

paper_content:

This paper presents a 3D approach for recognizing faces based on Principal Component Analysis (PCA). The approach addresses the issue of proper 3D face alignment required by PCA for maximum data compression and good generalization performance for new untrained faces. This issue has traditionally been addressed by 2D data normalization, a step that eliminates 3D object size information important for the recognition process. We achieve correspondence of facial points by registering a 3D face to a scaled generic 3D reference face and subsequently perform a surface normal search algorithm. 3D scaling of the generic reference face is performed to enable better alignment of facial points while preserving important 3D size information in the input face. The benefits of this approach for 3D face recognition and dimensionality reduction have been demonstrated on components of the Face Recognition Grand Challenge (FRGC) database versions 1 and 2.
---
paper_title: Fusion of Summation Invariants in 3D Human Face Recognition

paper_content:

A novel family of 2D and 3D geometrically invariant features, called summation invariants is proposed for the recognition of the 3D surface of human faces. Focusing on a rectangular region surrounding the nose of a 3D facial depth map, a subset of the so called semi-local summation invariant features is extracted. Then the similarity between a pair of 3D facial depth maps is computed to determine whether they belong to the same person. Out of many possible combinations of these set of features, we select, through careful experimentation, a subset of features that yields best combined performance. Tested with the 3D facial data from the on-going Face Recognition Grand Challenge v1.0 dataset, the proposed new features exhibit significant performance improvement over the baseline algorithm distributed with the datase
---
paper_title: Three-Dimensional Face Recognition

paper_content:

An expression-invariant 3D face recognition approach is presented. Our basic assumption is that facial expressions can be modelled as isometries of the facial surface. This allows to construct expression-invariant representations of faces using the bending-invariant canonical forms approach. The result is an efficient and accurate face recognition algorithm, robust to facial expressions, that can distinguish between identical twins (the first two authors). We demonstrate a prototype system based on the proposed algorithm and compare its performance to classical face recognition methods.The numerical methods employed by our approach do not require the facial surface explicitly. The surface gradients field, or the surface metric, are sufficient for constructing the expression-invariant representation of any given face. It allows us to perform the 3D face recognition task while avoiding the surface reconstruction stage.
---
paper_title: An Expression Deformation Approach to Non-rigid 3D Face Recognition

paper_content:

The accuracy of non-rigid 3D face recognition approaches is highly influenced by their capacity to differentiate between the deformations caused by facial expressions from the distinctive geometric attributes that uniquely characterize a 3D face, interpersonal disparities. We present an automatic 3D face recognition approach which can accurately differentiate between expression deformations and interpersonal disparities and hence recognize faces under any facial expression. The patterns of expression deformations are first learnt from training data in PCA eigenvectors. These patterns are then used to morph out the expression deformations. Similarity measures are extracted by matching the morphed 3D faces. PCA is performed in such a way it models only the facial expressions leaving out the interpersonal disparities. The approach was applied on the FRGC v2.0 dataset and superior recognition performance was achieved. The verification rates at 0.001 FAR were 98.35% and 97.73% for scans under neutral and non-neutral expressions, respectively.
---
paper_title: 3D FACE RECOGNITION BY POINT SIGNATURES AND ISO-CONTOURS

paper_content:

The paper addresses the problem of face recognition from range images. A novel technique based on matching of level contours is proposed and compared with a variant of the point signatures algorithm. Their efficiency is investigated on conditions of changes in expression and pose, and presence of glasses. Using a large database of range images, comparative experimental results are presented, showing that iso-contours outperformpoint signatures both in computational efficiency and in recognition rates.
---
paper_title: Three-Dimensional Face Recognition Using Shapes of Facial Curves

paper_content:

We study shapes of facial surfaces for the purpose of face recognition. The main idea is to 1) represent surfaces by unions of level curves, called facial curves, of the depth function and 2) compare shapes of surfaces implicitly using shapes of facial curves. The latter is performed using a differential geometric approach that computes geodesic lengths between closed curves on a shape manifold. These ideas are demonstrated using a nearest-neighbor classifier on two 3D face databases: Florida State University and Notre Dame, highlighting a good recognition performance
---
paper_title: Deformation Modeling for Robust 3D Face Matching

paper_content:

Face recognition based on 3D surface matching is promising for overcoming some of the limitations of current 2D image-based face recognition systems. The 3D shape is generally invariant to the pose and lighting changes, but not invariant to the nonrigid facial movement such as expressions. Collecting and storing multiple templates to account for various expressions for each subject in a large database is not practical. We propose a facial surface modeling and matching scheme to match 2.5D facial scans in the presence of both nonrigid deformations and pose changes (multiview) to a stored 3D face model with neutral expression. A hierarchical geodesic-based resampling approach is applied to extract landmarks for modeling facial surface deformations. We are able to synthesize the deformation learned from a small group of subjects (control group) onto a 3D neutral model (not in the control group), resulting in a deformed template. A user-specific (3D) deformable model is built for each subject in the gallery with respect to the control group by combining the templates with synthesized deformations. By fitting this generative deformable model to a test scan, the proposed approach is able to handle expressions and pose changes simultaneously. A fully automatic and prototypic deformable model based 3D face matching system has been developed. Experimental results demonstrate that the proposed deformation modeling scheme increases the 3D face matching accuracy in comparison to matching with 3D neutral models by 7 and 10 percentage points, respectively, on a subset of the FRGC v2.0 3D benchmark and the MSU multiview 3D face database with expression variations.
---
paper_title: Matching 2.5D face scans to 3D models

paper_content:

The performance of face recognition systems that use two-dimensional images depends on factors such as lighting and subject's pose. We are developing a face recognition system that utilizes three-dimensional shape information to make the system more robust to arbitrary pose and lighting. For each subject, a 3D face model is constructed by integrating several 2.5D face scans which are captured from different views. 2.5D is a simplified 3D (x,y,z) surface representation that contains at most one depth value (z direction) for every point in the (x, y) plane. Two different modalities provided by the facial scan, namely, shape and texture, are utilized and integrated for face matching. The recognition engine consists of two components, surface matching and appearance-based matching. The surface matching component is based on a modified iterative closest point (ICP) algorithm. The candidate list from the gallery used for appearance matching is dynamically generated based on the output of the surface matching component, which reduces the complexity of the appearance-based matching stage. Three-dimensional models in the gallery are used to synthesize new appearance samples with pose and illumination variations and the synthesized face images are used in discriminant subspace analysis. The weighted sum rule is applied to combine the scores given by the two matching components. Experimental results are given for matching a database of 200 3D face models with 598 2.5D independent test scans acquired under different pose and some lighting and expression changes. These results show the feasibility of the proposed matching scheme.
---
paper_title: 2D and 3D face recognition: A survey

paper_content:

Government agencies are investing a considerable amount of resources into improving security systems as result of recent terrorist events that dangerously exposed flaws and weaknesses in today's safety mechanisms. Badge or password-based authentication procedures are too easy to hack. Biometrics represents a valid alternative but they suffer of drawbacks as well. Iris scanning, for example, is very reliable but too intrusive; fingerprints are socially accepted, but not applicable to non-consentient people. On the other hand, face recognition represents a good compromise between what's socially acceptable and what's reliable, even when operating under controlled conditions. In last decade, many algorithms based on linear/nonlinear methods, neural networks, wavelets, etc. have been proposed. Nevertheless, Face Recognition Vendor Test 2002 shown that most of these approaches encountered problems in outdoor conditions. This lowered their reliability compared to state of the art biometrics. This paper provides an ''ex cursus'' of recent face recognition research trends in 2D imagery and 3D model based algorithms. To simplify comparisons across different approaches, tables containing different collection of parameters (such as input size, recognition rate, number of addressed problems) are provided. This paper concludes by proposing possible future directions.
---
paper_title: 3D Face Recognition using Mapped Depth Images

paper_content:

This paper addresses 3D face recognition from facial shape. Firstly, we present an effective method to automatically extract ROI of facial surface, which mainly depends on automatic detection of facial bilateral symmetry plane and localization of nose tip. Then we build a reference plane through the nose tip for calculating the relative depth values. Considering the non-rigid property of facial surface, the ROI is triangulated and parameterized into an isomorphic 2D planar circle, attempting to preserve the intrinsic geometric properties. At the same time the relative depth values are also mapped. Finally we perform eigenface on the mapped relative depth image. The entire scheme is insensitive to pose variance. The experiment using FRGC database v1.0 obtains the rank-1 identification score of 95%, which outperforms the result of the PCA base-line method by 4%, which demonstrates the effectiveness of our algorithm.
---
paper_title: Three-dimensional face recognition: an eigensurface approach

paper_content:

We evaluate a new approach to face recognition using a variety of surface representations of three-dimensional facial structure. Applying principal component analysis (PCA), we show that high levels of recognition accuracy can be achieved on a large database of 3D face models, captured under conditions that present typical difficulties to more conventional two-dimensional approaches. Applying a range of image processing techniques we identify the most effective surface representation for use in such application areas as security, surveillance, data compression and archive searching.
---
paper_title: 3 D Face recognition by ICP-based shape matching

paper_content:

In this paper, we propose a novel face recognition approach based on 2.5D/3D shape matching. While most of existing methods use facial intensity image, we aim to develop a method using three-dimensional information of the human face. This is the main innovation of our technology. In our approach, the 3D dimensional information is introduced in order to overcome classical face recognition problems which are pose, illumination and facial expression variations. The paradigm is to build a 3D face gallery using a laser-based scanner: the off-line stage. In the on-line stage, the recognition, we capture one 2.5D face model at any view point and with any facial expressions. Our processing allows the identification of the presented person by performing the captured model with all faces from the database. Here, the Iterative Closest Point-based matching algorithm provides the pose of the probe whereas the region-based metric provides a spatial deviation between the probe and each face from the gallery. In this metric, we calculate the global recognition score as a weighted sum of region-based distances already labelled as mimic or static regions. For automatic 3D face segmentation, we use an immersion version of watershed segmentation algorithm. This paper also presents some experiments in order to shown illumination, pose and facial expression compensations.
---
paper_title: Three-Dimensional Face Recognition in the Presence of Facial Expressions: An Annotated Deformable Model Approach

paper_content:

In this paper, we present the computational tools and a hardware prototype for 3D face recognition. Full automation is provided through the use of advanced multistage alignment algorithms, resilience to facial expressions by employing a deformable model framework, and invariance to 3D capture devices through suitable preprocessing steps. In addition, scalability in both time and space is achieved by converting 3D facial scans into compact metadata. We present our results on the largest known, and now publicly available, face recognition grand challenge 3D facial database consisting of several thousand scans. To the best of our knowledge, this is the highest performance reported on the FRGC v2 database for the 3D modality
---
paper_title: Robust 3D Face Recognition by Local Shape Difference Boosting

paper_content:

This paper proposes a new 3D face recognition approach, Collective Shape Difference Classifier (CSDC), to meet practical application requirements, i.e., high recognition performance, high computational efficiency, and easy implementation. We first present a fast posture alignment method which is self-dependent and avoids the registration between an input face against every face in the gallery. Then, a Signed Shape Difference Map (SSDM) is computed between two aligned 3D faces as a mediate representation for the shape comparison. Based on the SSDMs, three kinds of features are used to encode both the local similarity and the change characteristics between facial shapes. The most discriminative local features are selected optimally by boosting and trained as weak classifiers for assembling three collective strong classifiers, namely, CSDCs with respect to the three kinds of features. Different schemes are designed for verification and identification to pursue high performance in both recognition and computation. The experiments, carried out on FRGC v2 with the standard protocol, yield three verification rates all better than 97.9 percent with the FAR of 0.1 percent and rank-1 recognition rates above 98 percent. Each recognition against a gallery with 1,000 faces only takes about 3.6 seconds. These experimental results demonstrate that our algorithm is not only effective but also time efficient.
---
paper_title: An efficient 3D face recognition approach based on the fusion of novel local low-level features

paper_content:

We present a novel 3D face recognition approach based on low-level geometric features that are collected from the eyes-forehead and the nose regions. These regions are relatively less influenced by the deformations that are caused by facial expressions. The extracted features revealed to be efficient and robust in the presence of facial expressions. A region-based histogram descriptor computed from these features is used to uniquely represent a 3D face. A Support Vector Machine (SVM) is then trained as a classifier based on the proposed histogram descriptors to recognize any test face. In order to combine the contributions of the two facial regions (eyes-forehead and nose), both feature-level and score-level fusion schemes have been tested and compared. The proposed approach has been tested on FRGC v2.0 and BU-3DFE datasets through a number of experiments and a high recognition performance was achieved. Based on the results of ''neutral vs. non-neutral'' experiment of FRGC v2.0 and ''low-intensity vs. high-intensity'' experiment of BU-3DFE, the feature-level fusion scheme achieved verification rates of 97.6% and 98.2% at 0.1% False Acceptance Rate (FAR) and identification rates of 95.6% and 97.7% on the two datasets respectively. The experimental results also have shown that the feature-level fusion scheme outperformed the score-level fusion one.
---
paper_title: Face Recognition Based on Depth and Curvature Features

paper_content:

Face recognition from a representation based on features extracted from range images is explored. Depth and curvature features have several advantages over more traditional intensity-based features. Specifically, curvature descriptors have the potential for higher accuracy in describing surface-based events, are better suited to describe properties of the face in areas such as the cheeks, forehead, and chin, and are viewpoint invariant. Faces are represented in terms of a vector of feature descriptors. Comparisons between two faces is made based on their relationship in the feature space. The author provides a detailed analysis of the accuracy and discrimination of the particular features extracted, and the effectiveness of the recognition system for a test database of 24 faces. Recognition rates are in the range of 80% to 100%. In many cases, feature accuracy is limited more by surface resolution than by the extraction process. >
---
paper_title: 3D Face Recognition under Expressions, Occlusions, and Pose Variations

paper_content:

We propose a novel geometric framework for analyzing 3D faces, with the specific goals of comparing, matching, and averaging their shapes. Here we represent facial surfaces by radial curves emanating from the nose tips and use elastic shape analysis of these curves to develop a Riemannian framework for analyzing shapes of full facial surfaces. This representation, along with the elastic Riemannian metric, seems natural for measuring facial deformations and is robust to challenges such as large facial expressions (especially those with open mouths), large pose variations, missing parts, and partial occlusions due to glasses, hair, and so on. This framework is shown to be promising from both-empirical and theoretical-perspectives. In terms of the empirical evaluation, our results match or improve upon the state-of-the-art methods on three prominent databases: FRGCv2, GavabDB, and Bosphorus, each posing a different type of challenge. From a theoretical perspective, this framework allows for formal statistical inferences, such as the estimation of missing facial parts using PCA on tangent spaces and computing average shapes.
---
paper_title: Three-dimensional model based face recognition

paper_content:

The performance of face recognition systems that use two-dimensional (2D) images is dependent on consistent conditions such as lighting, pose and facial expression. We are developing a multi-view face recognition system that utilizes three-dimensional (3D) information about the face to make the system more robust to these variations. This work describes a procedure for constructing a database of 3D face models and matching this database to 2.5D face scans which are captured from different views, using coordinate system invariant properties of the facial surface. 2.5D is a simplified 3D (x, y, z) surface representation that contains at most one depth value (z direction) for every point in the (x, y) plane. A robust similarity metric is defined for matching, based on an iterative closest point (ICP) registration process. Results are given for matching a database of 18 3D face models with 113 2.5D face scans.
---
paper_title: Three-Dimensional Face Reconstruction From a Single Image by a Coupled RBF Network

paper_content:

Reconstruction of a 3-D face model from a single 2-D face image is fundamentally important for face recognition and animation because the 3-D face model is invariant to changes of viewpoint, illumination, background clutter, and occlusions. Given a coupled training set that contains pairs of 2-D faces and the corresponding 3-D faces, we train a novel coupled radial basis function network (C-RBF) to recover the 3-D face model from a single 2-D face image. The C-RBF network explores: 1) the intrinsic representations of 3-D face models and those of 2-D face images; 2) mappings between a 3-D face model and its intrinsic representation; and 3) mappings between a 2-D face image and its intrinsic representation. Since a particular face can be reconstructed by its nearest neighbors, we can assume that the linear combination coefficients for a particular 2-D face image reconstruction are identical to those for the corresponding 3-D face model reconstruction. Therefore, we can reconstruct a 3-D face model by using a single 2-D face image based on the C-RBF network. Extensive experimental results on the BU3D database indicate the effectiveness of the proposed C-RBF network for recovering the 3-D face model from a single 2-D face image.
---
paper_title: Two-dimensional PCA: a new approach to appearance-based face representation and recognition

paper_content:

In this paper, a new technique coined two-dimensional principal component analysis (2DPCA) is developed for image representation. As opposed to PCA, 2DPCA is based on 2D image matrices rather than 1D vectors so the image matrix does not need to be transformed into a vector prior to feature extraction. Instead, an image covariance matrix is constructed directly using the original image matrices, and its eigenvectors are derived for image feature extraction. To test 2DPCA and evaluate its performance, a series of experiments were performed on three face image databases: ORL, AR, and Yale face databases. The recognition rate across all trials was higher using 2DPCA than PCA. The experimental results also indicated that the extraction of image features is computationally more efficient using 2DPCA than PCA.
---
paper_title: Summation invariant and its applications to shape recognition

paper_content:

A novel summation invariant of curves under transformation group action is proposed. This new invariant is less sensitive to noise than the differential invariant and does not require an analytical expression for the curve as the integral invariant does. We exploit this summation invariant to define a shape descriptor called a semi-local summation invariant and use it as a new feature for shape recognition. Tested on a database of noisy shapes of fish, it was observed that the summation invariant feature exhibited superior discriminating power compared to that of wavelet-based invariant features.
---
paper_title: 3D human face recognition using point signature

paper_content:

We present a novel face recognition algorithm based on the point signature-a representation for free-form surfaces. We treat the face recognition problem as a non-rigid object recognition problem. The rigid parts of the face of one person are extracted after registering the range data sets of faces having different facial expressions. These rigid parts are used to create a model library for efficient indexing. For a test face, models are indexed from the library and the most appropriate models are ranked according to their similarity with the test face. Verification of each model face can be quickly and efficiently identified. Experimental results with range data involving six human subjects, each with four different facial expressions, have demonstrated the validity and effectiveness of our algorithm.
---
paper_title: Face Recognition Using Optimal Linear Components Of Range Images

paper_content:

This paper investigates the use of range images of faces for recognizing people. 3D scans of faces lead to range images that are linearly projected to low-dimensional subspaces for use in a classifier, say a nearest neighbor classifier or a support vector machine, to label people. Learning of subspaces is performed using an optimal component analysis, i.e. a stochastic optimization algorithm (on a Grassmann manifold) to find a subspace that maximizes classifier performance on the training image set. Results are presented for face recognition using FSU face database, and are compared with standard component anlyses such as PCA and ICA. This provides an efficient tool for analyzing certain aspects of facial shapes while avoiding a difficult task of geometric surface modeling.
---
paper_title: Curvature based human face recognition using depth weighted Hausdorff distance

paper_content:

In this paper, we propose a novel implementation of a person verification system based on depth-weighted Hausdorff distance (DWHD) using the surface curvatures of the human face. This new method incorporates the depth information and curvatures of local facial features. The weighting function used in this paper is based on depth values, which have differential properties of a face according to the people, so that the distance of this extracted edge maps are emphasized. Experimental results based on the combination of the maximum, minimum, and Gaussian curvature according to threshold values show that DWHD achieves recognition rate of 92.8%, 97.6% and 92.8% of the cases for 5 ranked candidates, respectively, and the proposed method of combined recognition rate for each curvature shows the best.
---
paper_title: Fast 3D face recognition based on normal map

paper_content:

This paper presents a 3D face recognition method aimed to biometric applications. The proposed method compares any two faces represented as 3D polygonal surfaces through their corresponding normal map, a bidimensional array which stores local curvature (mesh normals) as the pixel's RGB components of a color image. The recognition approach, based on the computation of a difference map resulting from the comparison of normal maps, is simple yet fast and accurate. A weighting mask, automatically generated for each subject using a set of expression variations, improves the robustness to a broad range of facial expressions. First results show the effectiveness of the method on a database of 3D faces featuring different genders, ages and expressions.
---
paper_title: Face recognition in hyperspectral images

paper_content:

Hyperspectral cameras provide useful discriminants for human face recognition that cannot be obtained by other imaging methods. We examine the utility of using near-infrared hyperspectral images for the recognition of faces over a database of 200 subjects. The hyperspectral images were collected using a CCD camera equipped with a liquid crystal tunable filter. Spectral measurements over the near-infrared allow the sensing of subsurface tissue structure, which is significantly different from person to person but relatively stable over time. The local spectral properties of human tissue are nearly invariant to face orientation and expression, which allows hyperspectral discriminants to be used for recognition over a large range of poses and expressions. We describe a face recognition algorithm that exploits spectral measurements for multiple facial tissue types. We demonstrate experimentally that this algorithm can be used to recognize faces over time in the presence of changes in facial pose and expression.
---
paper_title: Face Recognition in the Dark

paper_content:

Previous research has established thermal infrared imagery of faces as a valid biometric and has shown high recognition performance in a wide range of scenarios. However, all these results have been obtained using eye locations that were either manually marked, or automatically detected in a coregistered visible image, making the realistic use of thermal infrared imagery alone impossible. In this paper we present the results of an eye detector on thermal infrared imagery and we analyze its impact on recognition performance. Our experiments show that although eyes cannot be detected as reliably in thermal images as in visible ones, some face recognition algorithms can still achieve adequate performance.
---
paper_title: Illumination Invariant Face Recognition Using Near-Infrared Images

paper_content:

Most current face recognition systems are designed for indoor, cooperative-user applications. However, even in thus-constrained applications, most existing systems, academic and commercial, are compromised in accuracy by changes in environmental illumination. In this paper, we present a novel solution for illumination invariant face recognition for indoor, cooperative-user applications. First, we present an active near infrared (NIR) imaging system that is able to produce face images of good condition regardless of visible lights in the environment. Second, we show that the resulting face images encode intrinsic information of the face, subject only to a monotonic transform in the gray tone; based on this, we use local binary pattern (LBP) features to compensate for the monotonic transform, thus deriving an illumination invariant face representation. Then, we present methods for face recognition using NIR images; statistical learning algorithms are used to extract most discriminative features from a large pool of invariant LBP features and construct a highly accurate face matching engine. Finally, we present a system that is able to achieve accurate and fast face recognition in practice, in which a method is provided to deal with specular reflections of active NIR lights on eyeglasses, a critical issue in active NIR image-based face recognition. Extensive, comparative results are provided to evaluate the imaging hardware, the face and eye detection algorithms, and the face recognition algorithms and systems, with respect to various factors, including illumination, eyeglasses, time lapse, and ethnic groups
---
paper_title: Short Wavelength Infrared Face Recognition for Personalization

paper_content:

The paper describes an application of practical technologies to implement a low cost, consumer grade, single chip biometric system based on face recognition using infra-red imaging. The paper presents a system that consists of three stages that contribute in the face detection and recognition process. Each stage is explained with its individual contribution alongside results of tests performed for that stage. The system shows a high recognition rate when full frontal face images are fed to the system. The paper further discusses the application based approach in the automotive world with plans for further study. Recognition rates of the overall system are also presented.
---
paper_title: Discriminant Analysis for Recognition of Human Face Images

paper_content:

In this paper the discriminatory power of various human facial features is studied and a new scheme for Automatic Face Recognition (AFR) is proposed. Using Linear Discriminant Analysis (LDA) of different aspects of human faces in spatial domain, we first evaluate the significance of visual information in different parts/features of the face for identifying the human subject. The LDA of faces also provides us with a small set of features that carry the most relevant information for classification purposes. The features are obtained through eigenvector analysis of scatter matrices with the objective of maximizing between-class and minimizing within-class variations. The result is an efficient projection-based feature extraction and classification scheme for AFR. Soft decisions made based on each of the projections are combined, using probabilistic or evidential approaches to multisource data analysis. For medium-sized databases of human faces, good classification accuracy is achieved using very low-dimensional feature vectors.
---
paper_title: Infrared Face Recognition Using Distance Transforms

paper_content:

In this work we present an efficient approach for face recognition in the infrared spectrum. In the proposed approach physiological features are extracted from thermal images in order to build a unique thermal faceprint. Then, a distance transform is used to get an invariant representation for face recognition. The obtained physiological features are related to the distribution of blood vessels under the face skin. This blood network is unique to each individual and can be used in infrared face recognition. The obtained results are promising and show the effectiveness of the proposed scheme. Keywords—Face recognition, biometrics, infrared imaging. I. INTRODUCTION
---
paper_title: Illumination invariant face recognition using thermal infrared imagery

paper_content:

A key problem for face recognition has been accurate identification under variable illumination conditions. Conventional video cameras sense reflected light so that image grayvalues are a product of both intrinsic skin reflectivity and external incident illumination, thus obfuscating the intrinsic reflectivity of skin. Thermal emission from skin, on the other hand, is an intrinsic measurement that can be isolated from external illumination. We examine the invariance of Long-Wave InfraRed (LWIR) imagery with respect to different illumination conditions from the viewpoint of performance comparisons of two well-known face recognition algorithms applied to LWIR and visible imagery. We develop rigourous data collection protocols that formalize face recognition analysis for computer vision in the thermal IR.
---
paper_title: Multimodal face recognition: combination of geometry with physiological information

paper_content:

It is becoming increasingly important to be able to credential and identify authorized personnel at key points of entry. Such identity management systems commonly employ biometric identifiers. In this paper, we present a novel multimodal facial recognition approach that employs data from both visible spectrum and thermal infrared sensors. Data from multiple cameras is used to construct a three-dimensional mesh representing the face and a facial thermal texture map. An annotated face model with explicit two-dimensional parameterization (UV) is then fitted to this data to construct: 1) a three-channel UV deformation image encoding geometry, and 2) a one-channel UV vasculature image encoding facial vasculature. Recognition is accomplished by comparing: 1) the parametric deformation images, 2) the parametric vasculature images, and 3) the visible spectrum texture maps. The novelty of our work lies in the use of deformation images and physiological information as means for comparison. We have performed extensive tests on the Face Recognition Grand Challenge v1.0 dataset and on our own multimodal database with very encouraging results.
---
paper_title: Face recognition with visible and thermal infrared imagery

paper_content:

We present a comprehensive performance study of multiple appearance-based face recognition methodologies, on visible and thermal infrared imagery. We compare algorithms within the same imaging modality as well as between them. Both identification and verification scenarios are considered, and appropriate performance statistics reported for each case. Our experimental design is aimed at gaining full understanding of algorithm performance under varying conditions, and is based on Monte Carlo analysis of performance measures. This analysis reveals that under many circumstances, using thermal infrared imagery yields higher performance, while in other cases performance in both modalities is equivalent. Performance increases further when algorithms on visible and thermal infrared imagery are fused. Our study also provides a partial explanation for the multiple contradictory claims in the literature regarding performance of various algorithms on visible data sets.
---
paper_title: Infrared face recognition by using blood perfusion data

paper_content:

This paper presents a blood perfusion model of human faces based on thermodynamics and thermal physiology. The target is to convert the facial temperature data which are liable to ambient temperature into consistent blood perfusion data in order to improve the performance of infrared (IR) face recognition. Our large number of experiments has demonstrated that the blood perfusion data are less sensitive to ambient temperature if the human bodies are in steady state, and the real data testing demonstrated that the performance by means of blood perfusion data is significantly superior to that via temperature data in terms of recognition rate.
---
paper_title: Physiology-Based Face Recognition in the Thermal Infrared Spectrum

paper_content:

The current dominant approaches to face recognition rely on facial characteristics that are on or over the skin. Some of these characteristics have low permanency can be altered, and their phenomenology varies significantly with environmental factors (e.g., lighting). Many methodologies have been developed to address these problems to various degrees. However, the current framework of face recognition research has a potential weakness due to its very nature. We present a novel framework for face recognition based on physiological information. The motivation behind this effort is to capitalize on the permanency of innate characteristics that are under the skin. To establish feasibility, we propose a specific methodology to capture facial physiological patterns using the bioheat information contained in thermal imagery. First, the algorithm delineates the human face from the background using the Bayesian framework. Then, it localizes the superficial blood vessel network using image morphology. The extracted vascular network produces contour shapes that are characteristic to each individual. The branching points of the skeletonized vascular network are referred to as thermal minutia points (TMPs) and constitute the feature database. To render the method robust to facial pose variations, we collect for each subject to be stored in the database five different pose images (center, midleft profile, left profile, midright profile, and right profile). During the classification stage, the algorithm first estimates the pose of the test image. Then, it matches the local and global TMP structures extracted from the test image with those of the corresponding pose images in the database. We have conducted experiments on a multipose database of thermal facial images collected in our laboratory, as well as on the time-gap database of the University of Notre Dame. The good experimental results show that the proposed methodology has merit, especially with respect to the problem of low permanence over time. More importantly, the results demonstrate the feasibility of the physiological framework in face recognition and open the way for further methodological and experimental research in the area
---
paper_title: Highly Accurate and Fast Face Recognition Using Near Infrared Images

paper_content:

In this paper, we present a highly accurate, realtime face recognition system for cooperative user applications. The novelties are: (1) a novel design of camera hardware, and (2) a learning based procedure for effective face and eye detection and recognition with the resulting imagery. The hardware minimizes environmental lighting and delivers face images with frontal lighting. This avoids many problems in subsequent face processing to a great extent. The face detection and recognition algorithms are based on a local feature representation. Statistical learning is applied to learn most effective features and classifiers for building face detection and recognition engines. The novel imaging system and the detection and recognition engines are integrated into a powerful face recognition system. Evaluated in real-world user scenario, a condition that is harder than a technology evaluation such as Face Recognition Vendor Tests (FRVT), the system has demonstrated excellent accuracy, speed and usability.
---
paper_title: Face Recognition in the Thermal Infrared Spectrum

paper_content:

We present a two-stage face recognition method based on infrared imaging and statistical modeling. In the first stage we reduce the search space by finding highly likely candidates before arriving at a singular conclusion during the second stage. Previous work has shown that Bessel forms model accurately the marginal densities of filtered components and can be used to find likely matches but not a unique solution. We present an enhancement to this approach by applying Bessel modeling on the facial region only rather than the entire image and by pipelining a classification algorithm to produce a unique solution. The detailed steps of our method are as follows: First, the faces are separated from the background using adaptive fuzzy connectedness segmentation. Second, Gabor filtering is used as a spectral analysis tool. Third, the derivative filtered images are modeled using two-parameter Bessel forms. Fourth, high probability subjects are short-listed by applying the L^2 -norm on the Bessel models. Finally, the resulting set of highly likely matches is fed to a Bayesian classifier to find the exact match. We show experimentally that segmentation of the facial regions results in better hypothesis pruning and classification performance. We also present comparative experimental results with an eigenface approach to highlight the potential of our method.
---
paper_title: Automatic Feature Localization in Thermal Images for Facial Expression Recognition

paper_content:

We propose an unsupervised Local and Global feature extraction paradigm to approach the problem of facial expression recognition in thermal images. Starting from local, low-level features computed at interest point locations, our approach combines the localization of facial features with the holistic approach. The detailed steps are as follows: First, face localization using bi-modal thresholding is accomplished in order to localize facial features by way of a novel interest point detection and clustering approach. Second, we compute representative Eigenfeatures for feature extraction. Third, facial expression classification is made with a Support Vector Machine Committiee. Finally, the experiments over the IRIS data-set show that automation was achieved with good feature localization and classification performance.
---
paper_title: An Efficient Multimodal 2D-3D Hybrid Approach to Automatic Face Recognition

paper_content:

We present a fully automatic face recognition algorithm and demonstrate its performance on the FRGC v2.0 data. Our algorithm is multimodal (2D and 3D) and performs hybrid (feature based and holistic) matching in order to achieve efficiency and robustness to facial expressions. The pose of a 3D face along with its texture is automatically corrected using a novel approach based on a single automatically detected point and the Hotelling transform. A novel 3D spherical face representation (SFR) is used in conjunction with the scale-invariant feature transform (SIFT) descriptor to form a rejection classifier, which quickly eliminates a large number of candidate faces at an early stage for efficient recognition in case of large galleries. The remaining faces are then verified using a novel region-based matching approach, which is robust to facial expressions. This approach automatically segments the eyes- forehead and the nose regions, which are relatively less sensitive to expressions and matches them separately using a modified iterative closest point (ICP) algorithm. The results of all the matching engines are fused at the metric level to achieve higher accuracy. We use the FRGC benchmark to compare our results to other algorithms that used the same database. Our multimodal hybrid algorithm performed better than others by achieving 99.74 percent and 98.31 percent verification rates at a 0.001 false acceptance rate (FAR) and identification rates of 99.02 percent and 95.37 percent for probes with a neutral and a nonneutral expression, respectively.
---
paper_title: An evaluation of multimodal 2D+3D face biometrics

paper_content:

We report on the largest experimental study to date in multimodal 2D+3D face recognition, involving 198 persons in the gallery and either 198 or 670 time-lapse probe images. PCA-based methods are used separately for each modality and match scores in the separate face spaces are combined for multimodal recognition. Major conclusions are: 1) 2D and 3D have similar recognition performance when considered individually, 2) combining 2D and 3D results using a simple weighting scheme outperforms either 2D or 3D alone, 3) combining results from two or more 2D images using a similar weighting scheme also outperforms a single 2D image, and 4) combined 2D+3D outperforms the multi-image 2D result. This is the first (so far, only) work to present such an experimental control to substantiate multimodal performance improvement.
---
paper_title: Evaluation of automatic 4D face recognition using surface and texture registration

paper_content:

We introduce a novel technique for face recognition by using 4D face data that has been reconstructed from a stereo camera system. The 4D face data consists of a dense 3D mesh of vertices describing the facial geometry as well as a 2D texture map describing the facial appearance of each subject. The combination of geometry and texture information produces a complete photo-realistic model of each face. We propose a recognition algorithm based on two steps: The first step involves a 3D or 4D rigid registration of the faces. In the second step we introduce and evaluate different similarity metrics that measure the distance between pairs of closest points on two faces. A key advantage of the proposed technique is the fact that it can capture facial variations irrespective of the posture of the subject. We use this technique on 3D surface and texture data comprising 62 subjects at various postures and emotional expressions. Our results demonstrate that for subjects that look straight into the camera the recognition rate significantly increases when texture and geometry are combined in a 4D similarity metric.
---
paper_title: Facial feature detection and face recognition from 2D and 3D images

paper_content:

Abstract This paper presents a feature-based face recognition system based on both 3D range data as well as 2D gray-level facial images. Feature points are described by Gabor filter responses in the 2D domain and Point Signature in the 3D domain. Extracted shape features from 3D feature points and texture features from 2D feature points are first projected into their own subspace using PCA. In subspace, the corresponding normalized shape and texture weight vectors are then integrated to form an augmented vector which is used to represent each facial image. For a given test facial image, the best match in the model library is identified according to similarity function or Support Vector Machine (SVM). Experimental results involving 50 persons with different facial expressions and extracted from different viewpoints have demonstrated the efficiency of our algorithm.
---
paper_title: Spatially Optimized Data-Level Fusion of Texture and Shape for Face Recognition

paper_content:

Data-level fusion is believed to have the potential for enhancing human face recognition. However, due to a number of challenges, current techniques have failed to achieve its full potential. We propose spatially optimized data/pixel-level fusion of 3-D shape and texture for face recognition. Fusion functions are objectively optimized to model expression and illumination variations in linear subspaces for invariant face recognition. Parameters of adjacent functions are constrained to smoothly vary for effective numerical regularization. In addition to spatial optimization, multiple nonlinear fusion models are combined to enhance their learning capabilities. Experiments on the FRGC v2 data set show that spatial optimization, higher order fusion functions, and the combination of multiple such functions systematically improve performance, which is, for the first time, higher than score-level fusion in a similar experimental setup.
---
paper_title: Automatic 3D face recognition from depth and intensity Gabor features

paper_content:

As is well known, traditional 2D face recognition based on optical (intensity or color) images faces many challenges, such as illumination, expression, and pose variation. In fact, the human face generates not only 2D texture information but also 3D shape information. In this paper, we investigate what contributions depth and intensity information makes to face recognition when expression and pose variations are taken into account, and we propose a novel system for combining depth and intensity information to improve face recognition systems. In our system, local features described by Gabor wavelets are extracted from depth and intensity images, which are obtained from 3D data after fine alignment. Then a novel hierarchical selecting scheme embedded in linear discriminant analysis (LDA) and AdaBoost learning is proposed to select the most effective and most robust features and to construct a strong classifier. Experiments are performed on the CASIA 3D face database and the FRGC V2.0 database, two data sets with complex variations, including expressions, poses and long time lapses between two scans. Experimental results demonstrate the promising performance of the proposed method. In our system, all processes are performed automatically, thus providing a prototype of automatic face recognition combining depth and intensity information.
---
paper_title: Matching 2.5D face scans to 3D models

paper_content:

The performance of face recognition systems that use two-dimensional images depends on factors such as lighting and subject's pose. We are developing a face recognition system that utilizes three-dimensional shape information to make the system more robust to arbitrary pose and lighting. For each subject, a 3D face model is constructed by integrating several 2.5D face scans which are captured from different views. 2.5D is a simplified 3D (x,y,z) surface representation that contains at most one depth value (z direction) for every point in the (x, y) plane. Two different modalities provided by the facial scan, namely, shape and texture, are utilized and integrated for face matching. The recognition engine consists of two components, surface matching and appearance-based matching. The surface matching component is based on a modified iterative closest point (ICP) algorithm. The candidate list from the gallery used for appearance matching is dynamically generated based on the output of the surface matching component, which reduces the complexity of the appearance-based matching stage. Three-dimensional models in the gallery are used to synthesize new appearance samples with pose and illumination variations and the synthesized face images are used in discriminant subspace analysis. The weighted sum rule is applied to combine the scores given by the two matching components. Experimental results are given for matching a database of 200 3D face models with 598 2.5D independent test scans acquired under different pose and some lighting and expression changes. These results show the feasibility of the proposed matching scheme.
---
paper_title: Use of depth and colour eigenfaces for face recognition

paper_content:

In the present paper a face recognition technique is developed based on depth and colour information. The main objective of the paper is to evaluate three different approaches (colour, depth, combination of colour and depth) for face recognition and quantify the contribution of depth. The proposed face recognition technique is based on the implementation of the principal component analysis algorithm and the extraction of depth and colour eigenfaces. Experimental results show significant gains attained with the addition of depth information.
---
paper_title: Face localization and authentication using color and depth images

paper_content:

This paper presents a complete face authentication system integrating both two-dimensional (color or intensity) and three-dimensional (3-D) range data, based on a low-cost 3-D sensor, capable of real-time acquisition of 3-D and color images. Novel algorithms are proposed that exploit depth information to achieve robust face detection and localization under conditions of background clutter, occlusion, face pose alteration, and harsh illumination. The well-known embedded hidden Markov model technique for face authentication is applied to depth maps and color images. To cope with pose and illumination variations, the enrichment of face databases with synthetically generated views is proposed. The performance of the proposed authentication scheme is tested thoroughly on two distinct face databases of significant size. Experimental results demonstrate significant gains resulting from the combined use of depth and color or intensity information.
---
paper_title: Face Recognition Using 2D and 3D Facial Data

paper_content:

Results are presented for the largest experimental study to date that investigates the comparison and combination of 2D and 3D face recognition. To our knowledge, this is also the only such study to incorporate signicant time lapse between gallery and probe image acquisition, and to look at the effect of depth resolution. Recognition results are obtained in (1) single gallery and a single probe study, and (2) a single gallery and multiple probe study. A total of 275 subjects participated in one or more data acquisition sessions. Results are presented for gallery and probe datasets of 200 subjects imaged in both 2D and 3D, with one to thirteen weeks time lapse between gallery and probe images of a given subject yielding 951 pairs of 2D and 3D images. Using a PCA-based approach tuned separately for 2D and for 3D, we nd that 3D outperforms 2D. However, we also nd a multi-modal rank-one recognition rate of 98.5% in a single probe study and 98.8% in multi-probe study, which is statistically signicantly greater than either 2D or 3D alone.
---
paper_title: Passive Multimodal 2-D+3-D Face Recognition Using Gabor Features and Landmark Distances

paper_content:

We introduce a novel multimodal framework for face recognition based on local attributes calculated from range and portrait image pairs. Gabor coefficients are computed at automatically detected landmark locations and combined with powerful anthropometric features defined in the form of geodesic and Euclidean distances between pairs of fiducial points. We make the pragmatic assumption that the 2-D and 3-D data is acquired passively (e.g., via stereo ranging) with perfect registration between the portrait data and the range data. Statistical learning approaches are evaluated independently to reduce the dimensionality of the 2-D and 3-D Gabor coefficients and the anthropometric distances. Three parallel face recognizers that result from applying the best performing statistical learning schemes are fused at the match score-level to construct a unified multimodal (2-D+3-D) face recognition system with boosted performance. Performance of the proposed algorithm is evaluated on a large public database of range and portrait image pairs and found to perform quite well.
---
paper_title: 2D and 3D multimodal hybrid face recognition

paper_content:

We present a 2D and 3D multimodal hybrid face recognition algorithm and demonstrate its performance on the FRGC v1.0 data. We use hybrid (feature-based and holistic) matching for the 3D faces and a holistic matching approach on the 2D faces. Feature-based matching is performed by offline segmenting each 3D face in the gallery into three regions, namely the eyes-forehead, the nose and the cheeks. The cheeks are discarded to avoid facial expressions and hair. During recognition, each feature in the gallery is automatically matched, using a modified ICP algorithm, with a complete probe face. The holistic 3D and 2D face matching is performed using PCA. Individual matching scores are fused after normalization and the results are compared to the BEE baseline performances in order to provide some answers to the first three conjectures of the FRGC. Our multimodal hybrid algorithm substantially outperformed others by achieving 100% verification rate at 0.0006 FAR.
---
paper_title: Fuzzy fusion for face recognition

paper_content:

Face recognition based only on the visual spectrum is not accurate or robust enough to be used in uncontrolled environments. This paper describes a fusion of visible and infrared (IR) imagery for face recognition. In this paper, a scheme based on membership function and fuzzy integral is proposed to fuse information from the two modalities. Recognition rate is used to evaluate the fusion scheme. Experimental results show the scheme improves recognition performance substantially.
---
paper_title: Optimized Visual and Thermal Image Fusion for Efficient Face Recognition

paper_content:

Data fusion of thermal and visual images is a solution to overcome the drawbacks present in individual thermal and visual images. Data fusion using different approach is discussed and results are presented in this paper. Traditional fusion approaches don't produce useful results for face recognition. An optimized approach for face data fusion is developed which works for face data fusion equally well as for non-face images. This paper presents the implementation of Human face recognition system using proposed optimized data fusion of visual and thermal images. Gabor filtering technique, which extracts facial features, is used as a face recognition technique to test the effectiveness of the fusion techniques. It has been found that by using the proposed fusion technique Gabor filter can recognize face even with variable expressions and light intensities.
---
paper_title: PCA-based face recognition in infrared imagery: baseline and comparative studies

paper_content:

Techniques for face recognition generally fall into global and local approaches, with the principal component analysis (PCA) being the most prominent global approach. We use the PCA algorithm to study the comparison and combination of infrared and typical visible-light images for face recognition. We examine the effects of lighting change, facial expression change and passage of time between the gallery image and probe image. Experimental results indicate that when there is substantial passage of time (greater than one week) between the gallery and probe images, recognition from typical visible-light images may outperform that from infrared images. Experimental results also indicate that the combination of the two generally outperforms either one alone. This is the only study that we know of to focus on the issue of how passage of time affects infrared face recognition.
---
paper_title: Thermal face recognition in an operational scenario

paper_content:

We present results on the latest advances in thermal infrared face recognition, and its use in combination with visible imagery. Previous research by the authors has shown high performance under very controlled conditions, or questionable performance under a wider range of conditions. This paper shows results on the use of thermal infrared and visible imagery for face recognition in operational scenarios. In particular, we show performance statistics for outdoor face recognition and recognition across multiple sessions. Our results support the conclusion that face recognition performance with thermal infrared imagery is stable over multiple sessions, and that fusion of modalities increases performance. As measured by the number of images and number of subjects, this is the largest ever reported study on thermal face recognition.
---
paper_title: Fusion based approach for thermal and visible face recognition under pose and expresivity variation

paper_content:

Many existing works in face recognition are based solely on visible images. The use of bimodal systems based on visible and thermal images is seldom reported in face recognition, despite its advantage of combining the discriminative power of both modalities, under expressions or pose variations. In this paper, we investigate the combined advantages of thermal and visible face recognition on a Principal Component Analysis (PCA) induced feature space, with PCA applied on each spectrum, on a relatively new thermal/visible face database - OTCBVS, for large pose and expression variations. The recognition is done through k-nearest neighbors classification. Our findings confirm that the recognition results are improved by the aid of thermal images over the classical approaches on visible images alone, when a suitably chosen classifier score fusion is employed. We also propose a validation scheme for deriving the optimal fusion score between the two recognition modalities.
---
paper_title: Infrared and visible image fusion for face recognition

paper_content:

Considerable progress has been made in face recognition research ::: over the last decade especially with the development of powerful ::: models of face appearance (i.e., eigenfaces). Despite the variety ::: of approaches and tools studied, however, face recognition is not ::: accurate or robust enough to be deployed in uncontrolled ::: environments. Recently, a number of studies have shown that ::: infrared (IR) imagery offers a promising alternative to visible ::: imagery due to its relative insensitive to illumination changes. ::: However, IR has other limitations including that it is opaque to ::: glass. As a result, IR imagery is very sensitive to facial ::: occlusion caused by eyeglasses. In this paper, we propose fusing ::: IR with visible images, exploiting the relatively lower ::: sensitivity of visible imagery to occlusions caused by eyeglasses. ::: Two different fusion schemes have been investigated in this study: ::: (1) image-based fusion performed in the wavelet domain and, (2) ::: feature-based fusion performed in the eigenspace domain. In both ::: cases, we employ Genetic Algorithms (GAs) to find an optimum ::: strategy to perform the fusion. To evaluate and compare the ::: proposed fusion schemes, we have performed extensive recognition ::: experiments using the Equinox face dataset and the popular method ::: of eigenfaces. Our results show substantial improvements in ::: recognition performance overall, suggesting that the idea of ::: fusing IR with visible images for face recognition deserves ::: further consideration.
---
paper_title: Multimodal face recognition: combination of geometry with physiological information

paper_content:

It is becoming increasingly important to be able to credential and identify authorized personnel at key points of entry. Such identity management systems commonly employ biometric identifiers. In this paper, we present a novel multimodal facial recognition approach that employs data from both visible spectrum and thermal infrared sensors. Data from multiple cameras is used to construct a three-dimensional mesh representing the face and a facial thermal texture map. An annotated face model with explicit two-dimensional parameterization (UV) is then fitted to this data to construct: 1) a three-channel UV deformation image encoding geometry, and 2) a one-channel UV vasculature image encoding facial vasculature. Recognition is accomplished by comparing: 1) the parametric deformation images, 2) the parametric vasculature images, and 3) the visible spectrum texture maps. The novelty of our work lies in the use of deformation images and physiological information as means for comparison. We have performed extensive tests on the Face Recognition Grand Challenge v1.0 dataset and on our own multimodal database with very encouraging results.
---
paper_title: IR and visible light face recognition

paper_content:

This paper presents the results of several large-scale studies of face recognition employing visible-light and infrared (IR) imagery in the context of principal component analysis. We find that in a scenario involving time lapse between gallery and probe, and relatively controlled lighting, (1) PCA-based recognition using visible-light images outperforms PCA-based recognition using infrared images, (2) the combination of PCA-based recognition using visible-light and infrared imagery substantially outperforms either one individually. In a same-session scenario (i.e., near-simultaneous acquisition of gallery and probe images) neither modality is significantly better than the other. These experimental results reinforce prior research that employed a smaller data set, presenting a convincing argument that, even across a broad experimental spectrum, the behaviors enumerated above are valid and consistent.
---
paper_title: Optimum fusion of visual and thermal face images for recognition

paper_content:

In this paper one investigation has been done to find the optimum level of fusion to find a fused image from visual as well as thermal images. Because of the use of face recognition system in critical areas like, authenticating an authorized person in highly secured areas, investigation of criminals, online monitoring etc, face recognition system should be very robust and accurate one. This work is an attempt to fuse visual and thermal face images at optimum level to extract the advantages of visual as well as thermal images. In our work, Object Tracking and Classification Beyond Visible Spectrum (OTCBVS) database has been used for the visual and thermal images. Among all the experiments a maximum recognition result obtained is 93%.
---
paper_title: Physiology-Based Face Recognition in the Thermal Infrared Spectrum

paper_content:

The current dominant approaches to face recognition rely on facial characteristics that are on or over the skin. Some of these characteristics have low permanency can be altered, and their phenomenology varies significantly with environmental factors (e.g., lighting). Many methodologies have been developed to address these problems to various degrees. However, the current framework of face recognition research has a potential weakness due to its very nature. We present a novel framework for face recognition based on physiological information. The motivation behind this effort is to capitalize on the permanency of innate characteristics that are under the skin. To establish feasibility, we propose a specific methodology to capture facial physiological patterns using the bioheat information contained in thermal imagery. First, the algorithm delineates the human face from the background using the Bayesian framework. Then, it localizes the superficial blood vessel network using image morphology. The extracted vascular network produces contour shapes that are characteristic to each individual. The branching points of the skeletonized vascular network are referred to as thermal minutia points (TMPs) and constitute the feature database. To render the method robust to facial pose variations, we collect for each subject to be stored in the database five different pose images (center, midleft profile, left profile, midright profile, and right profile). During the classification stage, the algorithm first estimates the pose of the test image. Then, it matches the local and global TMP structures extracted from the test image with those of the corresponding pose images in the database. We have conducted experiments on a multipose database of thermal facial images collected in our laboratory, as well as on the time-gap database of the University of Notre Dame. The good experimental results show that the proposed methodology has merit, especially with respect to the problem of low permanence over time. More importantly, the results demonstrate the feasibility of the physiological framework in face recognition and open the way for further methodological and experimental research in the area
---
paper_title: Fusion of Visual and Thermal Signatures with Eyeglass Removal for Robust Face Recognition

paper_content:

This paper describes a fusion of visual and thermal infrared (IR) images for robust face recognition. Two types of fusion methods are discussed: data fusion and decision fusion. Data fusion produces an illumination-invariant face image by adaptively integrating registered visual and thermal face images. Decision fusion combines matching scores of individual face recognition modules. In the data fusion process, eyeglasses, which block thermal energy, are detected from thermal images and replaced with an eye template. Three fusion-based face recognition techniques are implemented and tested: Data fusion of visual and thermal images (Df), Decision fusion with highest matching score (Fh), and Decision fusion with average matching score (Fa). A commercial face recognition software FaceIt® is used as an individual recognition module. Comparison results show that fusion-based face recognition techniques outperformed individual visual and thermal face recognizers under illumination variations and facial expressions.
---
paper_title: Fusion of Thermal and Visual Images for efficient Face Recognition using Gabor Filter

paper_content:

Face recognition from visual images is difficult task due to illumination problem and in thermal imaging the main problem is of glasses. The solution of both these problem is data fusion of thermal and visual images. This paper presents the implementation of Human face recognition system using data fusion of visual and thermal images. Gabor filtering technique which extracts facial features is used in the proposed face recognition. . To our knowledge, this is the first visual and thermal data fusion recognition system which utilizes Gabor filter. Paper also discusses the performance improvement of face recognition along with issues of memory requirements.
---
paper_title: Facial Recognition Using Multisensor Images Based on Localized Kernel Eigen Spaces

paper_content:

A feature selection technique along with an information fusion procedure for improving the recognition accuracy of a visual and thermal image-based facial recognition system is presented in this paper. A novel modular kernel eigenspaces approach is developed and implemented on the phase congruency feature maps extracted from the visual and thermal images individually. Smaller sub-regions from a predefined neighborhood within the phase congruency images of the training samples are merged to obtain a large set of features. These features are then projected into higher dimensional spaces using kernel methods. The proposed localized nonlinear feature selection procedure helps to overcome the bottlenecks of illumination variations, partial occlusions, expression variations and variations due to temperature changes that affect the visual and thermal face recognition techniques. AR and Equinox databases are used for experimentation and evaluation of the proposed technique. The proposed feature selection procedure has greatly improved the recognition accuracy for both the visual and thermal images when compared to conventional techniques. Also, a decision level fusion methodology is presented which along with the feature selection procedure has outperformed various other face recognition techniques in terms of recognition accuracy.
---
paper_title: Overview of the face recognition grand challenge

paper_content:

Over the last couple of years, face recognition researchers have been developing new techniques. These developments are being fueled by advances in computer vision techniques, computer design, sensor design, and interest in fielding face recognition systems. Such advances hold the promise of reducing the error rate in face recognition systems by an order of magnitude over Face Recognition Vendor Test (FRVT) 2002 results. The face recognition grand challenge (FRGC) is designed to achieve this performance goal by presenting to researchers a six-experiment challenge problem along with data corpus of 50,000 images. The data consists of 3D scans and high resolution still imagery taken under controlled and uncontrolled conditions. This paper describes the challenge problem, data corpus, and presents baseline performance and preliminary results on natural statistics of facial imagery.
---
paper_title: Multimodal face recognition: combination of geometry with physiological information

paper_content:

It is becoming increasingly important to be able to credential and identify authorized personnel at key points of entry. Such identity management systems commonly employ biometric identifiers. In this paper, we present a novel multimodal facial recognition approach that employs data from both visible spectrum and thermal infrared sensors. Data from multiple cameras is used to construct a three-dimensional mesh representing the face and a facial thermal texture map. An annotated face model with explicit two-dimensional parameterization (UV) is then fitted to this data to construct: 1) a three-channel UV deformation image encoding geometry, and 2) a one-channel UV vasculature image encoding facial vasculature. Recognition is accomplished by comparing: 1) the parametric deformation images, 2) the parametric vasculature images, and 3) the visible spectrum texture maps. The novelty of our work lies in the use of deformation images and physiological information as means for comparison. We have performed extensive tests on the Face Recognition Grand Challenge v1.0 dataset and on our own multimodal database with very encouraging results.
---
paper_title: Fuzzy fusion for face recognition

paper_content:

Face recognition based only on the visual spectrum is not accurate or robust enough to be used in uncontrolled environments. This paper describes a fusion of visible and infrared (IR) imagery for face recognition. In this paper, a scheme based on membership function and fuzzy integral is proposed to fuse information from the two modalities. Recognition rate is used to evaluate the fusion scheme. Experimental results show the scheme improves recognition performance substantially.
---
paper_title: Evaluation of automatic 4D face recognition using surface and texture registration

paper_content:

We introduce a novel technique for face recognition by using 4D face data that has been reconstructed from a stereo camera system. The 4D face data consists of a dense 3D mesh of vertices describing the facial geometry as well as a 2D texture map describing the facial appearance of each subject. The combination of geometry and texture information produces a complete photo-realistic model of each face. We propose a recognition algorithm based on two steps: The first step involves a 3D or 4D rigid registration of the faces. In the second step we introduce and evaluate different similarity metrics that measure the distance between pairs of closest points on two faces. A key advantage of the proposed technique is the fact that it can capture facial variations irrespective of the posture of the subject. We use this technique on 3D surface and texture data comprising 62 subjects at various postures and emotional expressions. Our results demonstrate that for subjects that look straight into the camera the recognition rate significantly increases when texture and geometry are combined in a 4D similarity metric.
---
paper_title: Fusion based approach for thermal and visible face recognition under pose and expresivity variation

paper_content:

Many existing works in face recognition are based solely on visible images. The use of bimodal systems based on visible and thermal images is seldom reported in face recognition, despite its advantage of combining the discriminative power of both modalities, under expressions or pose variations. In this paper, we investigate the combined advantages of thermal and visible face recognition on a Principal Component Analysis (PCA) induced feature space, with PCA applied on each spectrum, on a relatively new thermal/visible face database - OTCBVS, for large pose and expression variations. The recognition is done through k-nearest neighbors classification. Our findings confirm that the recognition results are improved by the aid of thermal images over the classical approaches on visible images alone, when a suitably chosen classifier score fusion is employed. We also propose a validation scheme for deriving the optimal fusion score between the two recognition modalities.
---
paper_title: Optimum fusion of visual and thermal face images for recognition

paper_content:

In this paper one investigation has been done to find the optimum level of fusion to find a fused image from visual as well as thermal images. Because of the use of face recognition system in critical areas like, authenticating an authorized person in highly secured areas, investigation of criminals, online monitoring etc, face recognition system should be very robust and accurate one. This work is an attempt to fuse visual and thermal face images at optimum level to extract the advantages of visual as well as thermal images. In our work, Object Tracking and Classification Beyond Visible Spectrum (OTCBVS) database has been used for the visual and thermal images. Among all the experiments a maximum recognition result obtained is 93%.
---
paper_title: A 3D Face Model for Pose and Illumination Invariant Face Recognition

paper_content:

Generative 3D face models are a powerful tool in computer vision. They provide pose and illumination invariance by modeling the space of 3D faces and the imaging process. The power of these models comes at the cost of an expensive and tedious construction process, which has led the community to focus on more easily constructed but less powerful models. With this paper we publish a generative 3D shape and texture model, the Basel Face Model (BFM), and demonstrate its application to several face recognition task. We improve on previous models by offering higher shape and texture accuracy due to a better scanning device and less correspondence artifacts due to an improved registration algorithm. The same 3D face model can be fit to 2D or 3D images acquired under different situations and with different sensors using an analysis by synthesis method. The resulting model parameters separate pose, lighting, imaging and identity parameters, which facilitates invariant face recognition across sensors and data sets by comparing only the identity parameters. We hope that the availability of this registered face model will spur research in generative models. Together with the model we publish a set of detailed recognition and reconstruction results on standard databases to allow complete algorithm comparisons.
---
paper_title: Multimodal 2D, 2.5D & 3D Face Verification

paper_content:

A multimodal face verification process is presented for standard 2D color images, 2.5D range images and 3D meshes. A normalization in orientation and position is essential for 2.5D and 3D images to obtain a corrected frontal image. This is achieved using the spin images of the nose tip and both eyes, which feed an SVM classifier. First, a traditional principal component analysis followed by an SVM classifier are applied to both 2D and 2.5D images. Second, an iterative closest point algorithm is used to match 3D meshes. In all cases, the equal error rate is computed for different kinds of images in the training and test phases. In general, 2.5D range images show the best results (0.1% EER for frontal images). A special improvement in success rate for turned faces has been obtained for normalized 2.5D and 3D images compared to standard 2D images.
---
paper_title: SCface – surveillance cameras face database

paper_content:

In this paper we describe a database of static images of human faces. Images were taken in uncontrolled indoor environment using five video surveillance cameras of various qualities. Database contains 4,160 static images (in visible and infrared spectrum) of 130 subjects. Images from different quality cameras should mimic real-world conditions and enable robust face recognition algorithms testing, emphasizing different law enforcement and surveillance use case scenarios. In addition to database description, this paper also elaborates on possible uses of the database and proposes a testing protocol. A baseline Principal Component Analysis (PCA) face recognition algorithm was tested following the proposed protocol. Other researchers can use these test results as a control algorithm performance score when testing their own algorithms on this dataset. Database is available to research community through the procedure described at www.scface.org .
---
paper_title: A Natural Visible and Infrared Facial Expression Database for Expression Recognition and Emotion Inference

paper_content:

To date, most facial expression analysis has been based on visible and posed expression databases. Visible images, however, are easily affected by illumination variations, while posed expressions differ in appearance and timing from natural ones. In this paper, we propose and establish a natural visible and infrared facial expression database, which contains both spontaneous and posed expressions of more than 100 subjects, recorded simultaneously by a visible and an infrared thermal camera, with illumination provided from three different directions. The posed database includes the apex expressional images with and without glasses. As an elementary assessment of the usability of our spontaneous database for expression recognition and emotion inference, we conduct visible facial expression recognition using four typical methods, including the eigenface approach [principle component analysis (PCA)], the fisherface approach [PCA + linear discriminant analysis (LDA)], the Active Appearance Model (AAM), and the AAM-based + LDA. We also use PCA and PCA+LDA to recognize expressions from infrared thermal images. In addition, we analyze the relationship between facial temperature and emotion through statistical analysis. Our database is available for research purposes.
---
paper_title: Representation Plurality and Fusion for 3-D Face Recognition

paper_content:

In this paper, we present an extensive study of 3D face recognition algorithms and examine the benefits of various score-, rank-, and decision-level fusion rules. We investigate face recognizers from two perspectives: the data representation techniques used and the feature extraction algorithms that match best each representation type. We also consider novel applications of various feature extraction techniques such as discrete Fourier transform, discrete cosine transform, nonnegative matrix factorization, and principal curvature directions to the shape modality. We discuss and compare various classifier combination methods such as fixed rules and voting- and rank-based fusion schemes. We also present a dynamic confidence estimation algorithm to boost fusion performance. In identification experiments performed on FRGC v1.0 and FRGC v2.0 face databases, we have tried to find the answers to the following questions: 1) the relative importance of the face representation techniques vis-a-vis the types of features extracted; 2) the impact of the gallery size; 3) the conditions, under which subspace methods are preferable, and the compression factor; 4) the most advantageous fusion level and fusion methods; 5) the role of confidence votes in improving fusion and the style of selecting experts in the fusion; and 6) the consistency of the conclusions across different databases.
---
paper_title: An Efficient Multimodal 2D-3D Hybrid Approach to Automatic Face Recognition

paper_content:

We present a fully automatic face recognition algorithm and demonstrate its performance on the FRGC v2.0 data. Our algorithm is multimodal (2D and 3D) and performs hybrid (feature based and holistic) matching in order to achieve efficiency and robustness to facial expressions. The pose of a 3D face along with its texture is automatically corrected using a novel approach based on a single automatically detected point and the Hotelling transform. A novel 3D spherical face representation (SFR) is used in conjunction with the scale-invariant feature transform (SIFT) descriptor to form a rejection classifier, which quickly eliminates a large number of candidate faces at an early stage for efficient recognition in case of large galleries. The remaining faces are then verified using a novel region-based matching approach, which is robust to facial expressions. This approach automatically segments the eyes- forehead and the nose regions, which are relatively less sensitive to expressions and matches them separately using a modified iterative closest point (ICP) algorithm. The results of all the matching engines are fused at the metric level to achieve higher accuracy. We use the FRGC benchmark to compare our results to other algorithms that used the same database. Our multimodal hybrid algorithm performed better than others by achieving 99.74 percent and 98.31 percent verification rates at a 0.001 false acceptance rate (FAR) and identification rates of 99.02 percent and 95.37 percent for probes with a neutral and a nonneutral expression, respectively.
---
paper_title: Optimized Visual and Thermal Image Fusion for Efficient Face Recognition

paper_content:

Data fusion of thermal and visual images is a solution to overcome the drawbacks present in individual thermal and visual images. Data fusion using different approach is discussed and results are presented in this paper. Traditional fusion approaches don't produce useful results for face recognition. An optimized approach for face data fusion is developed which works for face data fusion equally well as for non-face images. This paper presents the implementation of Human face recognition system using proposed optimized data fusion of visual and thermal images. Gabor filtering technique, which extracts facial features, is used as a face recognition technique to test the effectiveness of the fusion techniques. It has been found that by using the proposed fusion technique Gabor filter can recognize face even with variable expressions and light intensities.
---
paper_title: Iterative Closest Normal Point for 3D Face Recognition

paper_content:

The common approach for 3D face recognition is to register a probe face to each of the gallery faces and then calculate the sum of the distances between their points. This approach is computationally expensive and sensitive to facial expression variation. In this paper, we introduce the iterative closest normal point method for finding the corresponding points between a generic reference face and every input face. The proposed correspondence finding method samples a set of points for each face, denoted as the closest normal points. These points are effectively aligned across all faces, enabling effective application of discriminant analysis methods for 3D face recognition. As a result, the expression variation problem is addressed by minimizing the within-class variability of the face samples while maximizing the between-class variability. As an important conclusion, we show that the surface normal vectors of the face at the sampled points contain more discriminatory information than the coordinates of the points. We have performed comprehensive experiments on the Face Recognition Grand Challenge database, which is presently the largest available 3D face database. We have achieved verification rates of 99.6 and 99.2 percent at a false acceptance rate of 0.1 percent for the all versus all and ROC III experiments, respectively, which, to the best of our knowledge, have seven and four times less error rates, respectively, compared to the best existing methods on this database.
---
paper_title: Illumination Invariant Face Recognition Using Near-Infrared Images

paper_content:

Most current face recognition systems are designed for indoor, cooperative-user applications. However, even in thus-constrained applications, most existing systems, academic and commercial, are compromised in accuracy by changes in environmental illumination. In this paper, we present a novel solution for illumination invariant face recognition for indoor, cooperative-user applications. First, we present an active near infrared (NIR) imaging system that is able to produce face images of good condition regardless of visible lights in the environment. Second, we show that the resulting face images encode intrinsic information of the face, subject only to a monotonic transform in the gray tone; based on this, we use local binary pattern (LBP) features to compensate for the monotonic transform, thus deriving an illumination invariant face representation. Then, we present methods for face recognition using NIR images; statistical learning algorithms are used to extract most discriminative features from a large pool of invariant LBP features and construct a highly accurate face matching engine. Finally, we present a system that is able to achieve accurate and fast face recognition in practice, in which a method is provided to deal with specular reflections of active NIR lights on eyeglasses, a critical issue in active NIR image-based face recognition. Extensive, comparative results are provided to evaluate the imaging hardware, the face and eye detection algorithms, and the face recognition algorithms and systems, with respect to various factors, including illumination, eyeglasses, time lapse, and ethnic groups
---
paper_title: 3D Face Recognition Using 3D Alignment for PCA

paper_content:

This paper presents a 3D approach for recognizing faces based on Principal Component Analysis (PCA). The approach addresses the issue of proper 3D face alignment required by PCA for maximum data compression and good generalization performance for new untrained faces. This issue has traditionally been addressed by 2D data normalization, a step that eliminates 3D object size information important for the recognition process. We achieve correspondence of facial points by registering a 3D face to a scaled generic 3D reference face and subsequently perform a surface normal search algorithm. 3D scaling of the generic reference face is performed to enable better alignment of facial points while preserving important 3D size information in the input face. The benefits of this approach for 3D face recognition and dimensionality reduction have been demonstrated on components of the Face Recognition Grand Challenge (FRGC) database versions 1 and 2.
---
paper_title: An Expression Deformation Approach to Non-rigid 3D Face Recognition

paper_content:

The accuracy of non-rigid 3D face recognition approaches is highly influenced by their capacity to differentiate between the deformations caused by facial expressions from the distinctive geometric attributes that uniquely characterize a 3D face, interpersonal disparities. We present an automatic 3D face recognition approach which can accurately differentiate between expression deformations and interpersonal disparities and hence recognize faces under any facial expression. The patterns of expression deformations are first learnt from training data in PCA eigenvectors. These patterns are then used to morph out the expression deformations. Similarity measures are extracted by matching the morphed 3D faces. PCA is performed in such a way it models only the facial expressions leaving out the interpersonal disparities. The approach was applied on the FRGC v2.0 dataset and superior recognition performance was achieved. The verification rates at 0.001 FAR were 98.35% and 97.73% for scans under neutral and non-neutral expressions, respectively.
---
paper_title: Spatially Optimized Data-Level Fusion of Texture and Shape for Face Recognition

paper_content:

Data-level fusion is believed to have the potential for enhancing human face recognition. However, due to a number of challenges, current techniques have failed to achieve its full potential. We propose spatially optimized data/pixel-level fusion of 3-D shape and texture for face recognition. Fusion functions are objectively optimized to model expression and illumination variations in linear subspaces for invariant face recognition. Parameters of adjacent functions are constrained to smoothly vary for effective numerical regularization. In addition to spatial optimization, multiple nonlinear fusion models are combined to enhance their learning capabilities. Experiments on the FRGC v2 data set show that spatial optimization, higher order fusion functions, and the combination of multiple such functions systematically improve performance, which is, for the first time, higher than score-level fusion in a similar experimental setup.
---
paper_title: Automatic 3D face recognition from depth and intensity Gabor features

paper_content:

As is well known, traditional 2D face recognition based on optical (intensity or color) images faces many challenges, such as illumination, expression, and pose variation. In fact, the human face generates not only 2D texture information but also 3D shape information. In this paper, we investigate what contributions depth and intensity information makes to face recognition when expression and pose variations are taken into account, and we propose a novel system for combining depth and intensity information to improve face recognition systems. In our system, local features described by Gabor wavelets are extracted from depth and intensity images, which are obtained from 3D data after fine alignment. Then a novel hierarchical selecting scheme embedded in linear discriminant analysis (LDA) and AdaBoost learning is proposed to select the most effective and most robust features and to construct a strong classifier. Experiments are performed on the CASIA 3D face database and the FRGC V2.0 database, two data sets with complex variations, including expressions, poses and long time lapses between two scans. Experimental results demonstrate the promising performance of the proposed method. In our system, all processes are performed automatically, thus providing a prototype of automatic face recognition combining depth and intensity information.
---
paper_title: 3D Face Recognition using Mapped Depth Images

paper_content:

This paper addresses 3D face recognition from facial shape. Firstly, we present an effective method to automatically extract ROI of facial surface, which mainly depends on automatic detection of facial bilateral symmetry plane and localization of nose tip. Then we build a reference plane through the nose tip for calculating the relative depth values. Considering the non-rigid property of facial surface, the ROI is triangulated and parameterized into an isomorphic 2D planar circle, attempting to preserve the intrinsic geometric properties. At the same time the relative depth values are also mapped. Finally we perform eigenface on the mapped relative depth image. The entire scheme is insensitive to pose variance. The experiment using FRGC database v1.0 obtains the rank-1 identification score of 95%, which outperforms the result of the PCA base-line method by 4%, which demonstrates the effectiveness of our algorithm.
---
paper_title: Three-Dimensional Face Recognition in the Presence of Facial Expressions: An Annotated Deformable Model Approach

paper_content:

In this paper, we present the computational tools and a hardware prototype for 3D face recognition. Full automation is provided through the use of advanced multistage alignment algorithms, resilience to facial expressions by employing a deformable model framework, and invariance to 3D capture devices through suitable preprocessing steps. In addition, scalability in both time and space is achieved by converting 3D facial scans into compact metadata. We present our results on the largest known, and now publicly available, face recognition grand challenge 3D facial database consisting of several thousand scans. To the best of our knowledge, this is the highest performance reported on the FRGC v2 database for the 3D modality
---
paper_title: Face recognition with visible and thermal infrared imagery

paper_content:

We present a comprehensive performance study of multiple appearance-based face recognition methodologies, on visible and thermal infrared imagery. We compare algorithms within the same imaging modality as well as between them. Both identification and verification scenarios are considered, and appropriate performance statistics reported for each case. Our experimental design is aimed at gaining full understanding of algorithm performance under varying conditions, and is based on Monte Carlo analysis of performance measures. This analysis reveals that under many circumstances, using thermal infrared imagery yields higher performance, while in other cases performance in both modalities is equivalent. Performance increases further when algorithms on visible and thermal infrared imagery are fused. Our study also provides a partial explanation for the multiple contradictory claims in the literature regarding performance of various algorithms on visible data sets.
---
paper_title: An efficient 3D face recognition approach based on the fusion of novel local low-level features

paper_content:

We present a novel 3D face recognition approach based on low-level geometric features that are collected from the eyes-forehead and the nose regions. These regions are relatively less influenced by the deformations that are caused by facial expressions. The extracted features revealed to be efficient and robust in the presence of facial expressions. A region-based histogram descriptor computed from these features is used to uniquely represent a 3D face. A Support Vector Machine (SVM) is then trained as a classifier based on the proposed histogram descriptors to recognize any test face. In order to combine the contributions of the two facial regions (eyes-forehead and nose), both feature-level and score-level fusion schemes have been tested and compared. The proposed approach has been tested on FRGC v2.0 and BU-3DFE datasets through a number of experiments and a high recognition performance was achieved. Based on the results of ''neutral vs. non-neutral'' experiment of FRGC v2.0 and ''low-intensity vs. high-intensity'' experiment of BU-3DFE, the feature-level fusion scheme achieved verification rates of 97.6% and 98.2% at 0.1% False Acceptance Rate (FAR) and identification rates of 95.6% and 97.7% on the two datasets respectively. The experimental results also have shown that the feature-level fusion scheme outperformed the score-level fusion one.
---
paper_title: A Comparative Study of 3-D Face Recognition Under Expression Variations

paper_content:

Research in face recognition has continuously been challenged by extrinsic (head pose, lighting conditions) and intrinsic (facial expression, aging) sources of variability. While many survey papers on face recognition exist, in this paper, we focus on a comparative study of 3-D face recognition under expression variations. As a first contribution, 3-D face databases with expressions are listed, and the most important ones are briefly presented and their complexity is quantified using the iterative closest point (ICP) baseline recognition algorithm. This allows to rank the databases according to their inherent difficulty for face-recognition tasks. This analysis reveals that the FRGC v2 database can be considered as the most challenging because of its size, the presence of expressions and outliers, and the time lapse between the recordings. Therefore, we recommend to use this database as a reference database to evaluate (expression-invariant) 3-D face-recognition algorithms. We also determine and quantify the most important factors that influence the performance. It appears that performance decreases 1) with the degree of nonfrontal pose, 2) for certain expression types, 3) with the magnitude of the expressions, 4) with an increasing number of expressions, and 5) for a higher number of gallery subjects. Future 3-D face-recognition algorithms should be evaluated on the basis of all these factors. As the second contribution, a survey of published 3-D face-recognition methods that deal with expression variations is given. These methods are subdivided into three classes depending on the way the expressions are handled. Region-based methods use expression-stable regions only, while other methods model the expressions either using an isometric or a statistical model. Isometric models assume the deformation because of expression variation to be (locally) isometric, meaning that the deformation preserves lengths along the surface. Statistical models learn how the facial soft tissue deforms during expressions based on a training database with expression labels. Algorithmic performances are evaluated by the comparison of recognition rates for identification and verification. No statistical significant differences in class performance are found between any pair of classes.
---
paper_title: Face localization and authentication using color and depth images

paper_content:

This paper presents a complete face authentication system integrating both two-dimensional (color or intensity) and three-dimensional (3-D) range data, based on a low-cost 3-D sensor, capable of real-time acquisition of 3-D and color images. Novel algorithms are proposed that exploit depth information to achieve robust face detection and localization under conditions of background clutter, occlusion, face pose alteration, and harsh illumination. The well-known embedded hidden Markov model technique for face authentication is applied to depth maps and color images. To cope with pose and illumination variations, the enrichment of face databases with synthetically generated views is proposed. The performance of the proposed authentication scheme is tested thoroughly on two distinct face databases of significant size. Experimental results demonstrate significant gains resulting from the combined use of depth and color or intensity information.
---
paper_title: Fusion of Thermal and Visual Images for efficient Face Recognition using Gabor Filter

paper_content:

Face recognition from visual images is difficult task due to illumination problem and in thermal imaging the main problem is of glasses. The solution of both these problem is data fusion of thermal and visual images. This paper presents the implementation of Human face recognition system using data fusion of visual and thermal images. Gabor filtering technique which extracts facial features is used in the proposed face recognition. . To our knowledge, this is the first visual and thermal data fusion recognition system which utilizes Gabor filter. Paper also discusses the performance improvement of face recognition along with issues of memory requirements.
---
paper_title: Facial Recognition Using Multisensor Images Based on Localized Kernel Eigen Spaces

paper_content:

A feature selection technique along with an information fusion procedure for improving the recognition accuracy of a visual and thermal image-based facial recognition system is presented in this paper. A novel modular kernel eigenspaces approach is developed and implemented on the phase congruency feature maps extracted from the visual and thermal images individually. Smaller sub-regions from a predefined neighborhood within the phase congruency images of the training samples are merged to obtain a large set of features. These features are then projected into higher dimensional spaces using kernel methods. The proposed localized nonlinear feature selection procedure helps to overcome the bottlenecks of illumination variations, partial occlusions, expression variations and variations due to temperature changes that affect the visual and thermal face recognition techniques. AR and Equinox databases are used for experimentation and evaluation of the proposed technique. The proposed feature selection procedure has greatly improved the recognition accuracy for both the visual and thermal images when compared to conventional techniques. Also, a decision level fusion methodology is presented which along with the feature selection procedure has outperformed various other face recognition techniques in terms of recognition accuracy.
---
paper_title: An Efficient Multimodal 2D-3D Hybrid Approach to Automatic Face Recognition

paper_content:

We present a fully automatic face recognition algorithm and demonstrate its performance on the FRGC v2.0 data. Our algorithm is multimodal (2D and 3D) and performs hybrid (feature based and holistic) matching in order to achieve efficiency and robustness to facial expressions. The pose of a 3D face along with its texture is automatically corrected using a novel approach based on a single automatically detected point and the Hotelling transform. A novel 3D spherical face representation (SFR) is used in conjunction with the scale-invariant feature transform (SIFT) descriptor to form a rejection classifier, which quickly eliminates a large number of candidate faces at an early stage for efficient recognition in case of large galleries. The remaining faces are then verified using a novel region-based matching approach, which is robust to facial expressions. This approach automatically segments the eyes- forehead and the nose regions, which are relatively less sensitive to expressions and matches them separately using a modified iterative closest point (ICP) algorithm. The results of all the matching engines are fused at the metric level to achieve higher accuracy. We use the FRGC benchmark to compare our results to other algorithms that used the same database. Our multimodal hybrid algorithm performed better than others by achieving 99.74 percent and 98.31 percent verification rates at a 0.001 false acceptance rate (FAR) and identification rates of 99.02 percent and 95.37 percent for probes with a neutral and a nonneutral expression, respectively.
---
paper_title: Thermal face recognition in an operational scenario

paper_content:

We present results on the latest advances in thermal infrared face recognition, and its use in combination with visible imagery. Previous research by the authors has shown high performance under very controlled conditions, or questionable performance under a wider range of conditions. This paper shows results on the use of thermal infrared and visible imagery for face recognition in operational scenarios. In particular, we show performance statistics for outdoor face recognition and recognition across multiple sessions. Our results support the conclusion that face recognition performance with thermal infrared imagery is stable over multiple sessions, and that fusion of modalities increases performance. As measured by the number of images and number of subjects, this is the largest ever reported study on thermal face recognition.
---
paper_title: An evaluation of multimodal 2D+3D face biometrics

paper_content:

We report on the largest experimental study to date in multimodal 2D+3D face recognition, involving 198 persons in the gallery and either 198 or 670 time-lapse probe images. PCA-based methods are used separately for each modality and match scores in the separate face spaces are combined for multimodal recognition. Major conclusions are: 1) 2D and 3D have similar recognition performance when considered individually, 2) combining 2D and 3D results using a simple weighting scheme outperforms either 2D or 3D alone, 3) combining results from two or more 2D images using a similar weighting scheme also outperforms a single 2D image, and 4) combined 2D+3D outperforms the multi-image 2D result. This is the first (so far, only) work to present such an experimental control to substantiate multimodal performance improvement.
---
paper_title: Illumination Invariant Face Recognition Using Near-Infrared Images

paper_content:

Most current face recognition systems are designed for indoor, cooperative-user applications. However, even in thus-constrained applications, most existing systems, academic and commercial, are compromised in accuracy by changes in environmental illumination. In this paper, we present a novel solution for illumination invariant face recognition for indoor, cooperative-user applications. First, we present an active near infrared (NIR) imaging system that is able to produce face images of good condition regardless of visible lights in the environment. Second, we show that the resulting face images encode intrinsic information of the face, subject only to a monotonic transform in the gray tone; based on this, we use local binary pattern (LBP) features to compensate for the monotonic transform, thus deriving an illumination invariant face representation. Then, we present methods for face recognition using NIR images; statistical learning algorithms are used to extract most discriminative features from a large pool of invariant LBP features and construct a highly accurate face matching engine. Finally, we present a system that is able to achieve accurate and fast face recognition in practice, in which a method is provided to deal with specular reflections of active NIR lights on eyeglasses, a critical issue in active NIR image-based face recognition. Extensive, comparative results are provided to evaluate the imaging hardware, the face and eye detection algorithms, and the face recognition algorithms and systems, with respect to various factors, including illumination, eyeglasses, time lapse, and ethnic groups
---
paper_title: An Expression Deformation Approach to Non-rigid 3D Face Recognition

paper_content:

The accuracy of non-rigid 3D face recognition approaches is highly influenced by their capacity to differentiate between the deformations caused by facial expressions from the distinctive geometric attributes that uniquely characterize a 3D face, interpersonal disparities. We present an automatic 3D face recognition approach which can accurately differentiate between expression deformations and interpersonal disparities and hence recognize faces under any facial expression. The patterns of expression deformations are first learnt from training data in PCA eigenvectors. These patterns are then used to morph out the expression deformations. Similarity measures are extracted by matching the morphed 3D faces. PCA is performed in such a way it models only the facial expressions leaving out the interpersonal disparities. The approach was applied on the FRGC v2.0 dataset and superior recognition performance was achieved. The verification rates at 0.001 FAR were 98.35% and 97.73% for scans under neutral and non-neutral expressions, respectively.
---
paper_title: Infrared and visible image fusion for face recognition

paper_content:

Considerable progress has been made in face recognition research ::: over the last decade especially with the development of powerful ::: models of face appearance (i.e., eigenfaces). Despite the variety ::: of approaches and tools studied, however, face recognition is not ::: accurate or robust enough to be deployed in uncontrolled ::: environments. Recently, a number of studies have shown that ::: infrared (IR) imagery offers a promising alternative to visible ::: imagery due to its relative insensitive to illumination changes. ::: However, IR has other limitations including that it is opaque to ::: glass. As a result, IR imagery is very sensitive to facial ::: occlusion caused by eyeglasses. In this paper, we propose fusing ::: IR with visible images, exploiting the relatively lower ::: sensitivity of visible imagery to occlusions caused by eyeglasses. ::: Two different fusion schemes have been investigated in this study: ::: (1) image-based fusion performed in the wavelet domain and, (2) ::: feature-based fusion performed in the eigenspace domain. In both ::: cases, we employ Genetic Algorithms (GAs) to find an optimum ::: strategy to perform the fusion. To evaluate and compare the ::: proposed fusion schemes, we have performed extensive recognition ::: experiments using the Equinox face dataset and the popular method ::: of eigenfaces. Our results show substantial improvements in ::: recognition performance overall, suggesting that the idea of ::: fusing IR with visible images for face recognition deserves ::: further consideration.
---
paper_title: Face recognition with temporal invariance: A 3D aging model

paper_content:

The variation caused by aging has not received adequate attention compared with pose, lighting, and expression variations. Aging is a complex process that affects both the 3D shape of the face and its texture (e.g., wrinkles). While the facial age modeling has been widely studied in computer graphics community, only a few studies have been reported in computer vision literature on age-invariant face recognition. We propose an automatic aging simulation technique that can assist any existing face recognition engine for aging-invariant face recognition. We learn the aging patterns of shape and the corresponding texture in 3D domain by adapting a 3D morphable model to the 2D aging database (public domain FG-NET). At recognition time, each probe and all gallery images are modified to compensate for the age-induced variation using an intermediate 3D model deformation and a texture modification, prior to matching. The proposed approach is evaluated on a set of age-separated probe and gallery data using a state-of-the-art commercial face recognition engine, FaceVACS. Use of 3D aging model improves the rank-1 matching accuracy on FG-NET database from 28.0% to 37.8%, on average.
---
paper_title: Multimodal face recognition: combination of geometry with physiological information

paper_content:

It is becoming increasingly important to be able to credential and identify authorized personnel at key points of entry. Such identity management systems commonly employ biometric identifiers. In this paper, we present a novel multimodal facial recognition approach that employs data from both visible spectrum and thermal infrared sensors. Data from multiple cameras is used to construct a three-dimensional mesh representing the face and a facial thermal texture map. An annotated face model with explicit two-dimensional parameterization (UV) is then fitted to this data to construct: 1) a three-channel UV deformation image encoding geometry, and 2) a one-channel UV vasculature image encoding facial vasculature. Recognition is accomplished by comparing: 1) the parametric deformation images, 2) the parametric vasculature images, and 3) the visible spectrum texture maps. The novelty of our work lies in the use of deformation images and physiological information as means for comparison. We have performed extensive tests on the Face Recognition Grand Challenge v1.0 dataset and on our own multimodal database with very encouraging results.
---
paper_title: IR and visible light face recognition

paper_content:

This paper presents the results of several large-scale studies of face recognition employing visible-light and infrared (IR) imagery in the context of principal component analysis. We find that in a scenario involving time lapse between gallery and probe, and relatively controlled lighting, (1) PCA-based recognition using visible-light images outperforms PCA-based recognition using infrared images, (2) the combination of PCA-based recognition using visible-light and infrared imagery substantially outperforms either one individually. In a same-session scenario (i.e., near-simultaneous acquisition of gallery and probe images) neither modality is significantly better than the other. These experimental results reinforce prior research that employed a smaller data set, presenting a convincing argument that, even across a broad experimental spectrum, the behaviors enumerated above are valid and consistent.
---
paper_title: 3-D Face Recognition Using eLBP-Based Facial Description and Local Feature Hybrid Matching

paper_content:

This paper presents an effective method for 3-D face recognition using a novel geometric facial representation along with a local feature hybrid matching scheme. The proposed facial surface description is based on a set of facial depth maps extracted by multiscale extended Local Binary Patterns (eLBP) and enables an efficient and accurate description of local shape changes; it thus enhances the distinctiveness of smooth and similar facial range images generated by preprocessing steps. The following matching strategy is SIFT-based and performs in a hybrid way that combines local and holistic analysis, robustly associating the keypoints between two facial representations of the same subject. As a result, the proposed approach proves robust to facial expression variations, partial occlusions, and moderate pose changes, and the last property makes our system registration-free for nearly frontal face models. The proposed method was experimented on three public datasets, i.e. FRGC v2.0, Gavab, and Bosphorus. It displays a rank-one recognition rate of 97.6% and a verification rate of 98.4% at a 0.001 FAR on the FRGC v2.0 database without any face alignment. Additional experiments on the Bosphorus dataset further highlight the advantages of the proposed method with regard to expression changes and external partial occlusions. The last experiment carried out on the Gavab database demonstrates that the entire system can also deal with faces under large pose variations and even partially occluded ones, when only aided by a coarse alignment process.
---
paper_title: Subject-Specific and Pose-Oriented Facial Features for Face Recognition Across Poses

paper_content:

Most face recognition scenarios assume that frontal faces or mug shots are available for enrollment to the database, faces of other poses are collected in the probe set. Given a face from the probe set, one needs to determine whether a match in the database exists. This is under the assumption that in forensic applications, most suspects have their mug shots available in the database, and face recognition aims at recognizing the suspects when their faces of various poses are captured by a surveillance camera. This paper considers a different scenario: given a face with multiple poses available, which may or may not include a mug shot, develop a method to recognize the face with poses different from those captured. That is, given two disjoint sets of poses of a face, one for enrollment and the other for recognition, this paper reports a method best for handling such cases. The proposed method includes feature extraction and classification. For feature extraction, we first cluster the poses of each subject's face in the enrollment set into a few pose classes and then decompose the appearance of the face in each pose class using Embedded Hidden Markov Model, which allows us to define a set of subject-specific and pose-priented (SSPO) facial components for each subject. For classification, an Adaboost weighting scheme is used to fuse the component classifiers with SSPO component features. The proposed method is proven to outperform other approaches, including a component-based classifier with local facial features cropped manually, in an extensive performance evaluation study.
---
paper_title: Physiology-Based Face Recognition in the Thermal Infrared Spectrum

paper_content:

The current dominant approaches to face recognition rely on facial characteristics that are on or over the skin. Some of these characteristics have low permanency can be altered, and their phenomenology varies significantly with environmental factors (e.g., lighting). Many methodologies have been developed to address these problems to various degrees. However, the current framework of face recognition research has a potential weakness due to its very nature. We present a novel framework for face recognition based on physiological information. The motivation behind this effort is to capitalize on the permanency of innate characteristics that are under the skin. To establish feasibility, we propose a specific methodology to capture facial physiological patterns using the bioheat information contained in thermal imagery. First, the algorithm delineates the human face from the background using the Bayesian framework. Then, it localizes the superficial blood vessel network using image morphology. The extracted vascular network produces contour shapes that are characteristic to each individual. The branching points of the skeletonized vascular network are referred to as thermal minutia points (TMPs) and constitute the feature database. To render the method robust to facial pose variations, we collect for each subject to be stored in the database five different pose images (center, midleft profile, left profile, midright profile, and right profile). During the classification stage, the algorithm first estimates the pose of the test image. Then, it matches the local and global TMP structures extracted from the test image with those of the corresponding pose images in the database. We have conducted experiments on a multipose database of thermal facial images collected in our laboratory, as well as on the time-gap database of the University of Notre Dame. The good experimental results show that the proposed methodology has merit, especially with respect to the problem of low permanence over time. More importantly, the results demonstrate the feasibility of the physiological framework in face recognition and open the way for further methodological and experimental research in the area
---
paper_title: A Comparative Study of 3-D Face Recognition Under Expression Variations

paper_content:

Research in face recognition has continuously been challenged by extrinsic (head pose, lighting conditions) and intrinsic (facial expression, aging) sources of variability. While many survey papers on face recognition exist, in this paper, we focus on a comparative study of 3-D face recognition under expression variations. As a first contribution, 3-D face databases with expressions are listed, and the most important ones are briefly presented and their complexity is quantified using the iterative closest point (ICP) baseline recognition algorithm. This allows to rank the databases according to their inherent difficulty for face-recognition tasks. This analysis reveals that the FRGC v2 database can be considered as the most challenging because of its size, the presence of expressions and outliers, and the time lapse between the recordings. Therefore, we recommend to use this database as a reference database to evaluate (expression-invariant) 3-D face-recognition algorithms. We also determine and quantify the most important factors that influence the performance. It appears that performance decreases 1) with the degree of nonfrontal pose, 2) for certain expression types, 3) with the magnitude of the expressions, 4) with an increasing number of expressions, and 5) for a higher number of gallery subjects. Future 3-D face-recognition algorithms should be evaluated on the basis of all these factors. As the second contribution, a survey of published 3-D face-recognition methods that deal with expression variations is given. These methods are subdivided into three classes depending on the way the expressions are handled. Region-based methods use expression-stable regions only, while other methods model the expressions either using an isometric or a statistical model. Isometric models assume the deformation because of expression variation to be (locally) isometric, meaning that the deformation preserves lengths along the surface. Statistical models learn how the facial soft tissue deforms during expressions based on a training database with expression labels. Algorithmic performances are evaluated by the comparison of recognition rates for identification and verification. No statistical significant differences in class performance are found between any pair of classes.
---
paper_title: 3D Face Recognition under Expressions, Occlusions, and Pose Variations

paper_content:

We propose a novel geometric framework for analyzing 3D faces, with the specific goals of comparing, matching, and averaging their shapes. Here we represent facial surfaces by radial curves emanating from the nose tips and use elastic shape analysis of these curves to develop a Riemannian framework for analyzing shapes of full facial surfaces. This representation, along with the elastic Riemannian metric, seems natural for measuring facial deformations and is robust to challenges such as large facial expressions (especially those with open mouths), large pose variations, missing parts, and partial occlusions due to glasses, hair, and so on. This framework is shown to be promising from both-empirical and theoretical-perspectives. In terms of the empirical evaluation, our results match or improve upon the state-of-the-art methods on three prominent databases: FRGCv2, GavabDB, and Bosphorus, each posing a different type of challenge. From a theoretical perspective, this framework allows for formal statistical inferences, such as the estimation of missing facial parts using PCA on tangent spaces and computing average shapes.
---
paper_title: A survey of approaches and challenges in 3D and multi-modal 3D + 2D face recognition

paper_content:

This survey focuses on recognition performed by matching models of the three-dimensional shape of the face, either alone or in combination with matching corresponding two-dimensional intensity images. Research trends to date are summarized, and challenges confronting the development of more accurate three-dimensional face recognition are identified. These challenges include the need for better sensors, improved recognition algorithms, and more rigorous experimental methodology.
---

Title: Recent Advances on Singlemodal and Multimodal Face Recognition: A Survey

Section 1: INTRODUCTION
Description 1: Introduce the goals and challenges of face recognition, and provide an overview of visual, 3D, and infrared face recognition methodologies.

Section 2: Visual Face Recognition
Description 2: Discuss the techniques and challenges associated with visual face recognition, highlighting the high performance achieved in controlled environments and the issues posed by varying factors such as illumination, pose, and expression.

Section 3: Three-Dimensional Face Recognition
Description 3: Explore the methods and benefits of 3D face recognition, including its robustness to pose and illumination variations, various approaches (feature-based and holistic), and the challenges posed by expression variations and data capture devices.

Section 4: Infrared Face Recognition
Description 4: Review the recent techniques of IR-based face recognition, categorized into statistical approaches and feature-based methods, and discuss the invariance of IR images to makeup and illumination changes.

Section 5: MULTIMODAL FACE RECOGNITION
Description 5: Examine the various multimodal face recognition techniques, including the fusion of visual, 3D, and infrared modalities, and discuss the benefits and challenges of combining multiple sensors and data types for improved recognition accuracy.

Section 6: FACE DATABASES AND PERFORMANCE EVALUATION
Description 6: Characterize the datasets used for training and testing face recognition methods, and summarize the performance of different methodologies across various databases, highlighting the importance of gallery and probe dataset sizes and variations.

Section 7: Discussion
Description 7: Provide a comprehensive discussion on the remaining challenges in face recognition, such as varying expressions, poses, and illumination, and the benefits and limitations of 3D and IR face recognition in addressing these issues.

Section 8: CONCLUSION
Description 8: Summarize the advances in face recognition, emphasizing the potential of multimodal techniques to overcome the limitations of individual modalities and offering insights on the future directions of face recognition research.