Three-dimensional face recognition using variance-based registration and subject-specific descriptors

Abstract

Face recognition underpins numerous applications; however, the task is still challenging mainly due to the variability of facial pose appearance. The existing methods show competitive performance but they are still short of what is needed. This article presents an effective three-dimensional pose invariant face recognition approach based on subject-specific descriptors. This results in state-of-the-art performance and delivers competitive accuracies. In our method, the face images are registered by transforming their acquisition pose into frontal view using three-dimensional variance of the facial data. The face recognition algorithm is initialized by detecting iso-depth curves in a coordinate plane perpendicular to the subject gaze direction. In this plane, discriminating keypoints are detected on the iso-depth curves of the facial manifold to define subject-specific descriptors using subject-specific regions. Importantly, the proposed descriptors employ Kernel Fisher Analysis-based features leading to the face recognition process. The proposed approach classifies unseen faces by pooling performance figures obtained from underlying classification algorithms. On the challenging data sets, FRGC v2.0 and GavabDB, our method obtains face recognition accuracies of 99.8% and 100% yielding superior performance compared to the existing methods.

Introduction

With the rapid advancement in three-dimensional (3-D) imaging and acquisition techniques, 3-D shape analysis and processing such as 3-D surface registration and shape retrieval have been extensively studied in the recent past. As a specific biometric application of 3-D shape analysis and processing, ¹ 3-D face recognition algorithms have also been largely evolved to identify individuals. Among other biometric modalities such as iris, fingerprint, and gait, the face is considered as an effective biometric due to noninvasive acquisition and social acceptance which does not involve subject’s cooperation. Human faces have been recognized using two-dimensional (2-D) images formerly, and existing commercial systems employ 2-D images at large scale. ² However; tackling the face pose, and illumination variations, is still a very challenging task. On the other hand, pose of the 3-D face images can be corrected by registering them in 3-D space. Likewise, they carry facial shape information and are resistant to variations in illumination. The main 3-D algorithms employ holistic or local features to recognize face images. The leading holistic feature-based algorithms are (i) principal component analysis (PCA), (ii) linear discriminant analysis (LDA), and (iii) independent component analysis, ³ whereas, major algorithms employing local features use (i) curves, ⁴ (ii) regions, ⁵ and (iii) points ² in the face recognition process.

The process of aligning facial shapes to a common coordinate system is called face registration. ⁶ It is the most fundamental and decisive step in a facial recognition system. Without registering the face images in a common coordinate system, the final recognition accuracy is severely degraded.

The objective of the proposed study is to realize (i) an efficient face registration algorithm based on variance of facial surfaces and (ii) a face recognition approach employing nine subject-specific descriptors (SSDs) and four parallel classification algorithms. The key innovations and contributions of the proposed algorithm are as follows:

The first contribution of the proposed study is based on the literature gap in the domain of 3-D face registration (“3-D face registration” section). The existing face registration techniques such as iterative closest point (ICP) and simulated annealing (SA) are computationally expensive, whereas, average face model (AFM) and intrinsic coordinate system (ICS) are less accurate. Conversely, the proposed study presents a novel variance-based coarse-to-fine 3-D face registration algorithm. Unlike the conventional face registration algorithms that register two face images with each other (such as ICP), the proposed algorithm is capable of registering a stand-alone face image. It rotates the probe face image and determines its variance for each rotation in xz, yz, and xy planes separately. The resulting variance is subsequently used to register the probe face image in frontal view. The proposed algorithm is computationally efficient than ICP and SA and more accurate than AFM and ICS.

The second contribution of the proposed approach is based on the following literature gaps in the research area of 3-D face recognition (“3D face recognition” section): (i) The existing algorithms did not propose any face registration algorithm and presented only face recognition algorithms. (ii) Some studies employed face registration algorithms which are computationally expensive. (iii) They did not employ SSDs. (iv) Their algorithm was tested on a single 3-D face database. On the other hand, the proposed approach is based on a novel and computationally efficient face registration algorithm and employs nine SSDs and four parallel classification algorithms. The recognition results obtained from the parallel classification algorithms are finally combined into a single decision using majority vote learning approach. It is observed that the proposed methodology returns excellent recognition accuracies. For performance evaluation of the proposed approach, the most commonly used 3-D face databases, namely, FRGC v2.0 ⁷ and GavabDB, ⁸ are employed in the experimentation process.

The rest of this study is organized as follows. A brief description of the related work is given in the second section. The proposed face registration and recognition methodology is presented in the third section. The experimental evaluation is reported in the fourth section. The fifth section presents results-related discussion whereas the sixth section concludes the findings and presents future work.

Literature review

This study consists of two components: (i) 3-D face registration and (ii) 3-D face recognition. Therefore, literature review of the two components is discussed separately.

3-D face registration

Existing 3-D face registration approaches are based on ICP, SA, AFM, and ICS.

The ICP ^{9 –11} -based registration approaches register a 3-D face image by rotating and translating it relative to the other such that mean squared error between them gets minimal. A significant drawback of the ICP-based registration is that it needs initial course alignment for convergence. Another disadvantage of this approach is that it is computationally expensive.

SA ¹² is a stochastic process-based registration algorithm, where centers of the masses of two faces are registered at first. Then a course registration is accomplished with a searching method by minimizing an estimation measure based on M-estimator SAmple Consensus and mean squared error of corresponding face points. Finally, a precise alignment is obtained by a searching algorithm using surface interpenetration measure-based estimation criterion. The downside of SA-based registration is its excessive computational time which is comparable to ICP-based registration approach.

AFM ¹³ -based registration approaches create an AFM by locating and averaging facial landmarks followed by the Procrustes analysis ¹⁴ and re-averaging of the resulting transformed landmark points. ¹⁵ Then the probe face image is registered to the AFM using ICP-based registration. A considerable weakness of the AFM-based registration is its low accuracy rates because some of the spatial information is lost while creating the AFM.

The ICS-based registration approaches register a face image by determining difference of angles between ICS and world coordinate system. The ICS is defined on the nose tip of the face image, whereas the world coordinate system exists along x-, y-, and z-axis. The resulting difference is used to correct the pose of the face image.

3-D face recognition

A comprehensive coverage of earlier proposed 3-D face recognition systems is presented in the literature. ^{16 –18} Some of the recent approaches are summarized as follows:

The study ¹⁹ proposed a local derivative pattern (LDP)-based approach for 3-D face recognition. The LDP acquires detailed facial information employing the local derivative variation in various directions. The algorithm was tested using FRGC v2.0 and Bosphorus databases producing 99.3% and 94.82% recognitions rates, respectively. This method can handle noisy images efficiently on its strength. On the other hand, one limitation of this method is ICP-based registration which is computationally extensive. Another limitation is parameter-based classification to produce high recognition rates which results into further increase of computational complexity.

The study ²⁰ proposed a depth image-based face descriptor employing regional covariance matrix based on PCA. The experimentation process was conducted using FRGC v2.0 and GavabDB databases resulting into 99.02% and 97.38% recognition rates, respectively. This study can overcome limitations of perspective distortion which is its advantage. Conversely, it did not employ any registration procedure which is a serious drawback. Another drawback of this study is the employment of PCA-based holistic technique which does not pay any particular attention to underlying class structure.

Another study ²¹ proposed a 3-D face recognition approach using local conformal parameterization and iso-geodesic stripes. The approach was tested using Texas3D and Bosphorus3D face databases obtaining 94% and 91% rank-1 recognition rates, respectively. This method carries the advantage of robustness to different head poses. On its downside, it is sensitive to occlusions in the nasal region which adversely affect the conformal parameterization.

A feature fusion-based face recognition approach is employed in the study ²² where the authors fused 2-D and 3-D facial features. The mentioned study utilized PCA along with canonical correlation analysis (CCA) to recognize a facial image, whereas the gallery images contained both of the 2-D and 3-D fused facial features. The authors conducted experiments on a database obtained from 115 subjects to classify neutral images employing Euclidean distance-based classifier and obtained 55% and 85% classification accuracy, using CCA and CCA-PCA algorithms in the respective order. The advantage of the study is its two-way feature extraction approach to improve recognition accuracies along with computational efficiency among contemporary methods. On the downside, the algorithm was tested only on a single already registered database and is unable to handle unregistered face images.

In another study proposed in Wang et al., ¹⁰ a score fusion-based 3-D face recognition algorithm is employed in which the authors extracted facial features using LDA and dual-tree complex wavelet transform (DT-CWT) from six facial regions. The classification process was implemented using nearest-neighbor (NN) matching criterion and sum rule. The method was tested on FRGC v2.0 database and 98.6% recognition rate was achieved using neutral face images. Sparse feature representation module of this study is beneficial in terms of computational efficiency, whereas the methodology did not address the registration issue on its downside.

The study ¹¹ proposed a 3-D face recognition method using radial face regions, covariance descriptors, and geodesic distance-based matching. The covariance-based descriptors enabled fusion of various features into a compact representation. The experimentation process yielded recognition rates of 99.2% and 100% considering all face images contained in the FRGC v2.0 and neutral subset of the GavabDB database, respectively. Covariance-based descriptors are beneficial in that they can be employed to fuse multiple heterogeneous features of arbitrary dimensions. Conversely, ICP-based registration is weakness of this study which increases the computational complexity. A recent study is proposed in Ganguly et al. ²³ equipped with an approach where edge curvature feature space is used by fusing edge and curvature maps. In this study, the faces were classified using fuzzy rule derived from Hausdorff distance vectors. In order to validate the algorithm, the authors conducted experiments on GavabDB and Frav3D databases achieving 87.20% and 98.87% recognition rates, respectively. An advantage of this study is that it is training-free, whereas its performance may suffer with increasing size of the facial database.

The study ²⁴ proposed an algorithm to detect seven landmark points on the face. This study is focused on employing feature points determined using Euclidean distance between pairs of the detected landmark points. The authors conducted experiments on Frav3D, GavabDB, and Bosphorus databases to validate their algorithm. Recognition rates of 75% and 87.5% were obtained on pose subset of Frav3D database using multilayer perceptron (MLP) and support vector machine (SVM), respectively. For GavabDB database, recognition rates of 62.5% and 87.5% were achieved on pose and expression subset using MLP and SVM, respectively. Similarly, recognition rates of 75% and 87.5% were obtained on pose, expression, and occlusion subset of Bosphorus database using MLP and SVM in the respective order. The study is beneficial because landmarks were localized without employing any registration procedure. On the other hand, due to landmark localization, the recognition rates can be significantly deteriorated in the presence of noisy images.

The circular region-based study ²⁵ proposed an effective 3-D face keypoint detection and matching framework using three principle curvature measures. The local shape of the face around each 3-D keypoint was comprehensively described by histograms of the principal curvature measures. Similarity comparison between facial surfaces was established by matching local shape descriptors using sparse representation-based reconstruction method and score level fusion. The evaluation of the algorithm was performed on GavabDB, FRGC v2.0, and Bosphorus databases obtaining 100% (neutral subset), 99.6% (neutral subset), and 98.6% (pose subset) recognition rates, respectively. This method is robust to head pose variations and occlusions, whereas involvement of facial landmarks that are prone to inaccuracy is a disadvantage.

Recently, a region-based face recognition approach is proposed in the study ²⁶ focused on detection of anthropometric landmarks. In this study, the intrinsic features of segmented regions are in turn used in the face recognition process to establish dense correspondences. Finally, a 3-D deformable model is created to be fitted to unseen face images to transfer the correspondences. The algorithm was tested on FRGC v2.0 and Bosphorus databases and recognition rates of 99.8% (neutral subset) and 95.7% (pose subset) were obtained using Cosine distance. The approach is beneficial as deformable model can be generated from scratch in contrast to other deformable model-based approaches. However, deformable model-based nature of the approach may be a cause of inaccuracy as compared to using original features.

An interesting study ²⁷ presented a facial character analysis algorithm based on Physiognomy science to interpret characteristic properties of a subject. Another study ²⁸ proposed a four-stage emotion analysis algorithm along with its application to NAO humanoid robot to understand instant emotional properties of a person.

The studies ^11,25,26 improved the face recognition performance by segmenting the face in 50 circular, nine circular, and five Voronoi regions, respectively. The proposed study is focused on recognizing the face images using nine SSDs which are classified using four separate classification algorithms. The results obtained by the proposed study are better than the state-of-the-art studies ^{4,10,25,29 –32} and ^{4,11,25,31,33 –35} using FRGC v2.0 and GavabDB databases, respectively.

Proposed 3-D face registration and recognition methodology

An overview of the proposed 3-D face registration and recognition approach is illustrated in Figure 1.

Figure 1.

Block diagram of the proposed system.

Variance-based 3-D face registration

In this study, a novel coarse-to-fine 3-D face registration algorithm is proposed based on variance of x, y, and z coordinate sets of the facial data distribution in xz, yz, and xy planes. The variances will be referred to as x-variance, y-variance, and z-variance, respectively, in the rest of the study. The variance of a random variable X in terms of expectation E is defined as its dispersion or spread about its mean m as given in equation (1) ³⁶

${\sigma }^2=E\left\{{\left(X-m\right)}^2\right\}$

1

The algorithm starts by localizing and cropping the face of a subject. In order to localize the face, nose tip heuristic is employed and is defined as the nearest point from 3-D scanner to the face image. Subsequently, the region of interest is cropped along x- and y-axis based on the nose tip coordinates. It is experimentally observed that when the face is rotated in xz plane in a circle, then x- and z-variances change sinusoidally in a complementary fashion (see Figure 2(a)), whereas, y-variance remains constant. In yz plane, y- and z-variances change sinusoidally in a complementary fashion (Figure 2(b)) and x-variance remains constant. Similarly, in xy plane, x- and y- variances change sinusoidally in a complementary fashion (Figure 2(c)) and z-variance remains constant. Please note that face is rotated in xz, yz, and xy planes using 3-D rotation matrix ³⁷ given below for registration into a common coordinate system

$A\hspace{0.17em}\text{=}\hspace{0.17em}\left[\begin{array}{ccc}\text{cos}\theta \text{cos}\varnothing & -\text{cos}\rho \text{sin}\varnothing \hspace{0.17em}\text{+}\hspace{0.17em}\text{sin}\rho \text{sin}\theta \text{cos}\varnothing & \text{sin}\rho \text{sin}\varnothing \text{+cos}\rho \text{sin}\theta \text{cos}\varnothing \\ \text{cos}\theta \text{sin}\varnothing & \text{cos}\rho \text{cos}\varnothing \hspace{0.17em}\text{+}\hspace{0.17em}\text{sin}\rho \text{sin}\theta \text{sin}\varnothing & -\text{sin}\rho \text{cos}\varnothing \text{+cos}\rho \text{sin}\theta \text{sin}\varnothing \\ -\text{sin}\theta & \text{sin}\rho \text{cosθ}& \text{cos}\rho \text{cos}\theta \end{array}\right]$

2

where angles θ, ρ, and φ correspond to rotations in xz, yz, and xy planes, respectively.

Figure 2.

Sinusoidally changing complementary variances in (a) xz plane, (b) yz plane, and (c) xy plane.

The insight behind the proposed registration approach is the human judgment-based observation that one of the two complementary variances corresponding to each of xz, yz, or xy planes separately gets maximal (while the other gets minimal) when a face accurately becomes in frontal pose in the respective planes. The face is rotated from 0° to 90° in xz plane, and x- and z-variances are computed. It is then rotated from 0° to 90° in yz plane to compute y- and z-variances. Similarly, x- and y-variances are computed by rotating the face from 0° to 90° in xy plane. Depending on the underlying facial data distribution in all cases, if the computed variance is increasing with facial rotations, the registration process gets based on rotation angle corresponding to the point of maximum variance (Figure 3(a) and (b)); otherwise, it is based on point of minimum variance in case of decreasing variances as shown in Figure 3(c). The maximum/minimum variance is referred to as designated variance in rest of the text. Please note that due to complementary behavior of the sinusoidal variances, this study employs x-variance in xz plane, y-variance in yz plane, and x-variance in xy plane because z, z, and y variances produce the same results in the respective planes. The implementation details of the proposed algorithm for each of xz, yz, and xy planes are given separately in the following section, whereas flow of the algorithm is presented in Figure 4.

Figure 3.

Designated variances computed at fine level using facial scan cara1_arriba from GavabDB database in (a) xz plane, (b) yz plane, and (c) xy plane.

Figure 4.

Flow diagram of the proposed methodology.

Registration in xz plane

The probe face image is registered in xz plane by computing x-variance using the procedure outlined below:

The probe face image is rotated in the range of −30° to +30° with a step size of +1°.

For each rotation, the corresponding x-variance is recorded to determine the designated variance.

The probe face image is coarsely registered by rotating it at a correction angle corresponding to the designated variance.

In order to register the probe face image at fine level, it is rotated in the range of −1° to +1° with a step size of 0.1°.

For each rotation, the corresponding x-variance is again noted to determine the designated variance.

Finally, the probe face image is registered at fine level by rotating it at a correction angle corresponding to the designated variance.

Registration in yz plane

The procedure to register the probe face image in yz plane is based on computing y-variance and is given as follows:

The probe face image is rotated in the range of −30° to +30° with a step size of +1°.

For each of the rotations, the corresponding y-variance is computed to determine the designated variance.

In order to coarsely register the probe face image, it is rotated at a correction angle corresponding to the designated variance.

For fine registration, the probe face image is rotated in the range of −1° to +1° with a step size of 0.1°.

To determine the designated variance, y-variance corresponding to each rotation is again noted.

The probe face image is finally rotated at a correction angle corresponding to the designated variance and is registered at fine level.

Registration in xy plane

The registration procedure in xy plane employs x-variance of the probe face image and is given as follows:

The probe face image is rotated in the range of −10° to +10° with a step size of 1°.

For each of the above-mentioned rotations, x-variance of the probe face image is computed to determine the designated variance.

The probe face image is registered in xy plane by rotating it at a correction angle corresponding to the designated variance.

Figure 5(a) to (c) and (d) to (f) represents the registration procedure for two example subjects from FRGC v2.0 and GavabDB databases. Example original and registered facial scans from FRGC v2.0 and GavabDB databases are shown in Figure 5(g) and (h), respectively.

Figure 5.

Registration of example face images between −30° and 30° (drawn between −90° and 90° to show variance trend): (a–c) FRGC v2.0: 02463d546; (d–f) GavabDB: cara2_arriba in (a, d) xz plane, (b, e) yz plane, and (c, f) xy plane; (g, h) example 3-D face images: original (row 1) and registered (row 2) from FRGC v2.0 and GavabDB databases, respectively.

The comparison of the registration algorithms was performed under equal conditions where the parameters of the algorithms were chosen proportionally to their optimal values. The optimal values of the registration parameters occur at 0° (at designated variances) in all of xz, yz, and xy planes separately, where face images have been turned into frontal position. The comparison procedure was carried out by performing statistical analysis using 50 randomly selected subjects from each of the FRGC v2.0 and GavabDB databases as follows:

The images were registered at optimal value of 0° in all of xz, yz, and xy planes separately.

Registration at optimal value of 0° was verified using our previous registration algorithm ³⁸ in all of xz, yz, and xy planes separately.

The images were then rotated between −90° and +90° with a step size of 1° and corresponding variances were noted in all three planes separately. Figure 6 represents the mean variance of randomly selected subjects, whereas maximum and minimum variances among the selected images at −25°, 0° and 50° are also shown in the same figure.

The statistical analysis clearly validates the proposed registration algorithm because the mean designated variances are observed at 0° in all three planes separately where face images become frontal.

Figure 6.

Statistical analysis of proposed registration algorithm for (a–c) FRGC v2.0 and (d–f) GavabDB in (a, d) xz plane, (b, e) yz plane, and (c, f) xy plane.

KFA-based subject-specific 3-D face recognition

Figure 4 represents flow of the proposed face recognition methodology. The proposed methodology is based on depth images ^17,19,21,26 which are generated by fitting a surface of the form z(x, y) to the 3-D point cloud employing the gridfit algorithm. ³⁹ A depth face image presents nose, eyes, forehead, and other facial features. It is comprised of a 2-D representation of a 3-D points set (x, y, z), where each pixel in the X–Y plane stores the depth value z. The proposed methodology employs iso-depth curves, keypoint extraction, subject-specific regions (SSRs), SSDs, and majority vote learning-based classification approach as follows.

Preprocessing: First of all, probe face image is preprocessed using median filtering to remove sharp spikes introduced during the face acquisition. Interpolation is then employed to fill the face image holes. In the end, low-pass filtering is used to smooth facial irregularities.

Iso-depth curve creation: An iso-depth curve (Figure 4) is created by slicing the probe face image in a plane P perpendicular to the gaze direction at a depth level (z = 50 pixels) from the nose tip. This curve describes most discriminative facial area around the nose. The iso-depth curve in plane P carries a specific intra-class pattern compared to inter-class patterns and is, therefore, subject-specific.

Keypoint extraction: In the plane P, x-axis and y-axis are drawn with origin defined at the nose tip.

• The nose tip is marked as the first keypoint c ₁.

• The intersection of positive and negative x-axis with iso-depth curve defines 2 keypoints c ₂ and c ₆.

• The intersection of positive and negative y-axis with the same curve defines another set of two keypoints c ₄ and $c_8$ .

• Rays $\overrightarrow{c_{1}p}$ , $\overrightarrow{c_{1}q}$ , $\overrightarrow{c_{1}r}$ , and $\overrightarrow{c_{1}s}$ are drawn from the nose tip c ₁ to points p, q, r, and s, respectively, such that they define angles $\angle p{c}_{1}x = {45}^{°}$ , $\angle q{c}_{1}x = {135}^{°}$ , $\angle r{c}_{1}x = {225}^{°}$ , and $\angle s{c}_{1}x = {315}^{°}$ . These rays intersect the iso-depth curve at points p, q, r, and s respectively, defining four more keypoints c ₃, c ₅, c ₇, and c ₉.

Consequently, a total of nine subject-specific keypoints (Figure 4) are determined along the locus of iso-depth curve and provide most discriminative facial information in the vicinity of the nose region.

SSRs: Once the keypoints have been determined, they are projected and marked on the facial surface to define nine overlapping circular regions R ₁–R ₉ each centered at the corresponding keypoint C ₁–C ₉.

• The radii of region R ₁ and each of remaining regions are ${\rho }_1$ and ${\rho }_2$ , respectively (Figure 4) and are set to satisfy the relation $\text{2 }{\rho }_{\text{2}}\text{ > }{\rho }_{\text{1}}$ for dense coverage of whole facial area.

• The size of the created feature vector for region R ₁ and each of the remaining regions R ₂ to R ₉ is 1600 and 625, respectively.

• The radii of SSRs are utilized to control the facial area defined by them for feature pooling. In general, smaller values of radii define smaller local surface descriptors which are robust to several facial variations, whereas too small values of radii decrease the discriminative information carried by the descriptor. ⁴⁰ . The proposed strategy of defining SSRs provides discriminating and stable facial features for comparison of diverse local variations and improves the accuracy of face recognition process.

• The circular regions create a quasi-daisy radial flower pattern as shown in Figure 4. Such a pattern of overlapping circles simulates the functioning of human complex cells in the visual cortex. ²⁵

• Because the circular regions R ₁–R ₉ are centered at subject-specific keypoints, they are subject-specific and referred to as SSRs. The SSRs can be readily observed for three subjects from FRGC v2.0 database in Figure 7. SSRs-based flower patterns are similar for intra-class face images, whereas they are different for inter-class face images (Figure 7).

KFA-based SSDs: SSDs (σ _i ) are built using Kernel Fisher Analysis (KFA) ⁴¹ -based facial features extracted from SSRs (R_i ) for i ∈ {1,2,3,…,9}. The objective of KFA is to yield a non-LDA by employing nonlinear mapping for projecting the input data into higher dimensional space. It seeks a nonlinear transformation matrix that can maximize the between-class scatter and minimize the within-class scatter. KFA reduces the feature vector space belonging to c different classes to a (c − 1) dimensional feature vector space. Thus, for FRGC v2.0 (c = 466) and GavabDB (c = 61) databases, the size of the feature vector turns out to be 465 and 60, respectively.

Classification: The SSDs ${\sigma }_{\text{i}}$ of the probe face image are matched to the gallery using four parallel classification algorithms which are based on Euclidean (Euc), Manhattan (Man), Cosine (Cos), and Mahalanobis Cosine (MahCos) distances summarized as follows.

• i. The Euc distance between two vectors m and n in image space is calculated by the formula given in equation (3). ⁴² The use of Euc distance is beneficial in face recognition-based classification problems as it exploits geometric features of the probe face image to enhance classification accuracy ⁴²

$D_{\text{EUC}}\left(\mathbf{m}\text{, }\mathbf{n}\right)\text{ = }\sqrt{{{\displaystyle \sum }}_{i\hspace{0.17em}\text{=}\hspace{0.17em}\text{1}}^N{\left({\mathbf{m}}_{\mathbf{i}} -{\mathbf{n}}_{\mathbf{i}}\right)}^{\text{2}}}$

3

• ii. The Man distance between two vectors m and n in image space is computed by the formula given in equation (4). ⁴³ The Man distance is robust to tackle the noise present in the facial images. It improves the performance of the face recognition system by classifying the noisy face images robustly ⁴³

$D_{\text{Man}}\left(\mathbf{m}\text{, }\mathbf{n}\right)\text{ =}{{\displaystyle \sum }}_{i\hspace{0.17em}\text{=}\hspace{0.17em}\text{1}}^N|{\mathbf{m}}_{\mathbf{i} }-{\mathbf{n}}_{\mathbf{i}}|$

4

• iii. The Cos distance between two vectors m and n in image space is defined in equation (5). ⁴² It has the advantage of utilizing angle-based correlation between face image vectors ⁴² to enhance the classification accuracies

$d_{\text{Cos}}\left(\mathbf{m}\text{,}\hspace{0.17em}\mathbf{n}\right)\text{ = }-\frac{{{\displaystyle \sum }}_{\mathrm{i}\hspace{0.17em}\text{=}\hspace{0.17em}\text{1}}^{\mathrm{N}}\left({\mathbf{m}}_{\mathbf{i}}{\mathbf{n}}_{\mathbf{i}}\right)}{\sqrt{{{\displaystyle \sum }}_{\mathrm{i}\hspace{0.17em}\text{=}\hspace{0.17em}\text{1}}^{\mathrm{N}}{\left({\mathbf{m}}_{\mathbf{i}}\right)}^{\text{2}}}\sqrt{{{\displaystyle \sum }}_{\mathrm{i}\hspace{0.17em}\text{=}\hspace{0.17em}\text{1}}^{\mathrm{N}}{\left({\mathbf{n}}_{\mathbf{i}}\right)}^{\text{2}}}}$

5

• Here, higher similarity reflects higher score and the final score is computed as given below

$D_{\text{Cos}}\left(\mathbf{m}\text{,}\hspace{0.17em}\mathbf{n}\right)\text{ = 1 }-d_{\text{Cos}}\left(\mathbf{m}\text{,}\hspace{0.17em}\mathbf{n}\right)$

6

• iv. MahCos distance is defined as Cosine distance determined in Mahalanobis space. The benefit of using MahCos distance is twofold: (i) it seeks to measure angle-based correlation between image vectors using Cosine distance and (ii) it takes into account covariance of image vectors in Mahalanobis space. ⁴² For two vectors represented by m and n in image space, MahCos distance is given in equations (7) and (8) as explained in Bajwa et al. ⁴²

$d_{\text{MahCos}}\left(\mathbf{m},\mathbf{n}\right)\text{ = }-\frac{{{\displaystyle \sum }}_{\mathrm{i}\hspace{0.17em}\text{=}\hspace{0.17em}\text{1}}^{\mathrm{N}}\left(\frac{{\mathbf{m}}_{\mathbf{i}}{\mathbf{n}}_{\mathbf{i}}}{{\sigma }_{\mathrm{i}}{\sigma }_{\mathrm{i}}}\right)}{\sqrt{{{\displaystyle \sum }}_{\mathrm{i}\hspace{0.17em}\text{=}\hspace{0.17em}\text{1}}^{\mathrm{N}}{\left(\frac{{\mathbf{m}}_{\mathbf{i}}}{{\sigma }_{\mathrm{i}}}\right)}^{\text{2}}}\sqrt{{{\displaystyle \sum }}_{\mathrm{i}\hspace{0.17em}\text{=}\hspace{0.17em}\text{1}}^{\mathrm{N}}{\left(\frac{{\mathbf{n}}_{\mathbf{i}}}{{\sigma }_{\mathrm{i}}}\right)}^{\text{2}}}}$

7

• where $\sigma i$ is defined as the standard deviation of ith dimension. Similar to Cos distance, the actual distance is computed by subtracting $d_{\text{MahCos}}$ from 1.

$D_{\text{MahCos}}\left(\mathbf{m},\mathbf{n}\right)\text{ =1}-d_{\text{MahCos}}\left(\mathbf{m},\mathbf{n}\right)$

8

• v. For each of the parallel classification algorithms, individual classification results obtained for each of SSD ${\sigma }_i$ are combined using weighted score level fusion defined as follows

$C_{i,j}\left(P\right)\text{ =}{\displaystyle {\sum }_{i,j}^{}{w}_{{\sigma }_{i,j}}{s}_{{\sigma }_{i,j}(P)}}$

9

• where ${\mathrm{C}}_{i,j}$ represents classification accuracies for probe image $P$ obtained using i SSDs and j classification algorithms, respectively. Similarly, $s_{{\sigma }_{i,j}}$ represents scores obtained for ith SSD using jth classification algorithm and $w_{{\sigma }_{i,j}}$ denotes corresponding weight assignments. The weights are computed using individual result accuracy obtained for each of ${\sigma }_i$ separately for each of the parallel classification algorithms as given in equation (10)

$w\text{ = }\frac{\text{Classifier accuracy for an individual }{\sigma }_i}{\sum \text{Individual classifier accuracies for all }{\sigma }_i}$

10

• The motivation behind using four classification algorithms in a fusion-based approach is that each of four parallel classification algorithms carries different benefits as mentioned in above paragraphs. Therefore, the classes misclassified by them do not necessarily overlap. Consequently, the final improved recognition accuracies are attributed to the properties of constituent parallel classification algorithms.

• vi. The classification accuracies yielded by the parallel classification algorithms for probe face image P are combined into a single classification decision $C\left(\text{P}\right)$ using majority vote learning approach as given in equation (11)

$C\left(P\right)\text{ = mode }\left\{C_{\text{Euc}}\left(P\right)\text{,}{C}_{\text{Man}}\left(P\right)\text{, }{C}_{\text{Cos}}\left(P\right)\text{,}{C}_{\text{MahCos}}\left(P\right)\right\}$

11

• where the mode function returns the most frequently yielded class by the parallel classification algorithms. In majority vote learning approach, each classification algorithm has right of one vote and the class obtaining more than half of votes is declared as a match. For “n” even number of classification algorithms, the majority is defined as $\frac{n}{2}+1$ . Therefore, three or four votes are required to decide for a true positive in the context of proposed approach as $n=4$ .

Figure 7.

SSRs for three subjects from FRGC v2.0. SSR: subject-specific region.

Experiments and results

In order to demonstrate the effectiveness of the proposed 3-D face recognition approach, the experimental evaluation is realized on two public 3-D face databases, namely, FRGC v2.0 ⁷ and GavabDB, ⁸ and results are given as rank-1 recognition rates. Explanation of the employed databases, experimental setup, and results are given as follows:

Experiments on FRGC v2.0 database

Database and experimental protocol description

FRGC v2.0 ⁷ database is comprised of 4007 images acquired from 466 subjects using Minolta Vivid 910 Laser scanner and is a license-based database. For this database, several experimentation settings have been designed by the providers. In the provided list of experiments, experiment 3 is designed to study the facial shape effects of the subjects. Among the subjects aged 18 years and above, nearly 60% carry neutral facial expression. The images have been captured at variable subject-scanner lengths with variable resolution and minimal variation in pose. Each of the 3-D image is given in terms of x, y, and z coordinates contained in three matrices of the size 480 × 640, whereas valid 3-D points of the three matrices are provided in the fourth matrix. Some of the face images from this database are shown in Figure 8.

Figure 8.

Example images from FRGC v2.0.

The performance of proposed recognition algorithm is evaluated by employing experimental protocol of Berretti et al. ²⁹ using FRGC v2.0 database consisting of 4007 face images acquired from 466 subjects. In order to conduct neutral versus neutral (N vs. N) face recognition experiment, the target set is comprised of first available neutral image of each subject or first stored image for each of few subjects which do not carry a neutral expression image in the database. Images contained in FRGC v1 are included in the training set as given in the database guidelines ²⁹ and 2469 neutral images are utilized as probe set. The performance figures are given in cumulative match characteristic (CMC) curves of Figure 9.

Figure 9.

CMC curves for N vs. N experiment on FRGC v2.0. CMC: cumulative match characteristic.

The above-mentioned experiment was also performed using 3-D mesh face model. Various generic 3-D mesh face models are available in the literature such as Candide and Waters face model. ⁴⁴ Candide model contains 113 vertices and 168 surfaces, whereas Waters model consists of 256 vertices and 441 surfaces. In this study, Candide face model is employed due to its simplicity.

In order to extract facial shape, the deformation procedure of Candide face model is employed in two phases: labeling and adaptation. In labeling phase, nine feature vertices are selected from the Candide model as shown in Figure 10. In the next step, 2-D projected positions of selected vertices are marked on the depth face image using a method similar to the study. ⁴⁴ In adaptation phase, vertices of the Candide model are iteratively adjusted employing propagation of feature vertex displacement. ⁴⁵ Finally, each model vertex is mapped with a depth value corresponding to the depth face image. The rank-1 recognition rates of 3-D mesh face model-based experiment are shown in Figure 11.

Figure 10.

Keypoints (small squares) and SSDs (in circles) created using Candide 3-D face model. SSD: subject-specific descriptor.

Figure 11.

Comparison with existing approaches using FRGC v2.0.

Comparative evaluation

Referring to Figure 11, the study ²⁹ employed iso-geodesic stripes and 3-D weighted walkthrough (3DWW) descriptors in the classification process. The study ³⁰ employed local features in 3-D face recognition using scale space extrema, score-based fusion and SVM classifier. The authors of the study ³¹ presented a 3-D face recognition approach employing multi-scale extended Local Binary Patterns descriptors and a local feature-based hybrid matching method. The study ⁴ proposed a curve-based approach using a Riemannian framework to analyze facial shapes using radial curves emanating from the nose tip. The authors in ¹⁰ focused on DT-CWT, LDA along with NN classifier. The study ³² focused on an approach to 3-D face recognition using Multiple Vertex-based description of Histogram of Multiple-order surface differential Quantities (MV-HOMQ) and fine-grained matcher. Finally, the study ²⁵ presented a 3-D face keypoint detection and matching approach based on principle curvatures where matching was performed using local shape descriptors, sparse representation-based reconstruction method and score level fusion. The proposed approach obtained 99.8% rank-1 recognition rate which is better than state-of-the art studies given in Figure 11. The improved recognition accuracy is attributed to the proposed accurate registration and SSDs-based face recognition using KFA.

Experiments on GavabDB database

Database and experimental protocol description

The GavabDB ⁸ database consists of 427 ³³ facial scans acquired from 61 Caucasian subjects (45 male and 16 female). Each subject is acquired nine times even with large pose and expression variations. The database contains four neutral scans among which two are frontal while the other two are looking up or down at nearly +35° and −35°, respectively. Three scans are captured with smile, laugh and a random expression while the remaining two are right and left profile scans acquired at +90° and −90° rotations in xz plane. Some of the facial scans from this database are presented in Figure 12.

Figure 12.

Example images from GavabDB.

In order to evaluate the performance of proposed algorithm on GavabDB database, three experiments are conducted using experimental protocol of Hariri et al., ¹¹ where first neutral image of each subject is included in the gallery and second neutral, rotated looking down and rotated looking up sets are used as probe sets in separate experiments. The rank-1 recognition rates for all three experimental settings are enlisted in the Table 1. The rank-1 recognition rates for the same three experiments employing 3-D mesh face model are also shown in the Table 1.

Table 1.

Comparison of rank-1 recognition rates using GavabDB database (%).

	Mahoor and Abdel Mottaleb 33	Huang et al. 31	Drira et al. 4	Berretti et al. 34	Tabia et al. 35	Hariri et al. 11	Tang et al. 25	Proposed methodology
								3-D mesh model based	Iso-depth curve based
Neutral	95	100	100	100	100	100	100	98.4	100
Rotated looking down	85.3	96.7	100	95.1	N/A	99.2a	98.4	96.7	98.4
Rotated looking up	88.6	96.7	98.4	96.7	N/A	98.4	100	98.4	100

^aTypo error, recognition rate can be 98.4 instead of 99.2 for 60 subjects recognized against a gallery of 61.

Comparative evaluation

Comparison of the proposed approach to the state-of-the-art earlier studies using GavabDB database is given in the Table 1. The study ³³ proposed a 3-D face recognition algorithm using 3-D binary ridge images, principal maximum curvature and ICP-based matching. The 3-D face recognition approaches ^4,25,31 are discussed in the “Comparative evaluation” section of “Experiments on FRGC v2.0 database” section. The methodology presented in the study ³⁴ is based on local descriptors extraction and matching using multiple scale keypoints detection according to differential surface measurements. The 3-D face recognition approach presented in the study ³⁵ employed covariance descriptors which capture geometric and spatial properties of a shape region along with their correlation. In this study, the matching was performed using geodesic distance. The 3-D face recognition algorithm ¹¹ encoded different types of features and modalities into a compact representation using covariance-based descriptors and employed geodesic distance in the matching process. The proposed approach yielded 100%, 98.4%, and 100% rank-1 recognition rates for neutral, rotated looking down, and rotated looking up subsets of the GavabDB database. The yielded recognition rates are better than state-of-the-art studies given in the Table 1. The enhanced recognition rates are attributed to the proposed accurate registration approach along with SSDs-based face recognition using KFA.

Computational complexity analysis

It is worthwhile to underline the two-phased nature of the proposed algorithm. The first phase is based on 3-D face registration which employs variance-based coarse-to-fine approach to register 3-D facial point clouds. In terms of Big-O notation the computational complexity of registration phase is O(N) which is linear with the shape size N. On the other hand, complexity of ICP-based algorithms is O(N ²) which is quadratic with the shape size N. ⁴⁶ The second phase consists of the face recognition module. For this phase, as the distance of each training image from the test image is to be computed in the classification stage, the complexity is linear and is of the order of O(N) where N is the size of the SSDs. The computational complexity analysis clearly reveals that the proposed 3-D face registration and recognition approach is computationally more efficient than the ICP-based face recognition algorithms.

Results-related discussion

The proposed study presents a novel 3-D face registration and recognition approach. Face registration module is based on 3-D variance of facial data while face recognition component employs KFA-based SSDs using nine SSRs and four parallel classification algorithms. The SSRs are extracted using keypoints determined at intersection points of a subject-specific iso-depth curve with x- and y-axis, and four rays drawn at 45°, 135°, 225°, and 315° angular positions. The experimental results demonstrated the following observations.

The 3-D variance-based registration approach successfully registered 3-D facial scans from FRGC v2.0 and GavabDB databases. The sinusoidal nature of the change in variance is attributed to rotation of the 3-D face image in a complete cycle. One complete counter-clock wise rotation of face image corresponds to 0° to 360° rotation as shown in the circular part of the Figure 13. This tallies a sinusoidal waveform ranging from 0° to 360°. More precisely, the face positions at 0° and 180° correspond to the front and back positions of the face, while the face positions at 90° and 270° represent the right and left profile views, respectively.

The statistical analysis clearly validated results of the proposed face registration approach.

Iso-depth curves are subject-specific and exhibit same intra-class and differing inter-class curve patterns. Therefore, the resulting keypoints and SSRs are also subject-specific and corresponding SSDs enhance the recognition accuracy.

The MahCos distance-based classification algorithm returned best individual rank-1 recognition rate than Euc, Man and Cos distance-based algorithms in the respective order.

The combined single classification decision outperformed the best individual SSR-based recognition rate.

The superior recognition rates yielded by iso-depth curve-based approach are attributed to subject-specific curve-based SSRs and SSDs which produced subject-specific flower patterns. On the other hand, lower recognition rates of 3-D mesh model-based approach are because of two facts: (i) The SSRs and SSDs were extracted on vertices of the Candide 3-D face model leading to fixed shape of flower pattern for all subjects, and therefore, are not as subject-specific and discriminative as those created using iso-depth curve-based approach. (ii) The facial features are needed to be deformed to fit the Candide model.

The proposed approach experimentally produced comparable results for both employed databases, and exhibited better performance than the approaches compared in Figure 11 and the Table 1.

Figure 13.

Sinusoidal nature of the change in variance.

Conclusion

A novel approach to 3-D face registration and recognition is contributed where images are registered using 3-D variance of the facial data. The recognition approach employed KFA-based facial features using SSDs based on nine SSRs extracted using keypoints determined on iso-depth curves. The SSDs are classified using four parallel classification algorithms. The proposed methodology obtained 99.8% rank-1 recognition rate using FRGC v2.0 database for N versus N experiment. Similarly rank-1 recognition rate of 100%, 98.4%, and 100% was achieved for neutral, rotated looking down, and rotated looking up subsets of the GavabDB database. In the future, local feature-based descriptors are intended to be studied using SSRs of variable sizes extracted at multiple iso-depth curves.