Enhanced Asymmetric Bilinear Model for Face Recognition

Abstract

Bilinear models have been successfully applied to separate two factors, for example, pose variances and different identities in face recognition problems. Asymmetric model is a type of bilinear model which models a system in the most concise way. But seldom there are works exploring the applications of asymmetric bilinear model on face recognition problem with illumination changes. In this work, we propose enhanced asymmetric model for illumination-robust face recognition. Instead of initializing the factor probabilities randomly, we initialize them with nearest neighbor method and optimize them for the test data. Above that, we update the factor model to be identified. We validate the proposed method on a designed data sample and extended Yale B dataset. The experiment results show that the enhanced asymmetric models give promising results and good recognition accuracies.

1. Introduction

Face recognition has been an active research area due to its applications in security entry systems and automatic law enforcement system. Pose variances, illumination changes, and noise from observations make it hard to define a model with promising recognition accuracies. Previous work [1–5] has proven that image variations of faces under varying illuminations could be modeled by linear spaces while view spaces which resulted from varying head poses are considered nonlinear [6]. Manifold learning [6, 7], bilinear models [8–10], linear dimensionality reduction [11, 12], and tensor analysis [13–15] have all been applied to solve face recognition problems. Recently deep learning techniques are also employed to solve face recognition problems and have achieved good performances [16–18].

Usually, faces need to be aligned before recognition. There has been lots of works which have been done on face alignment [19–30]. And it is well known that misalignment will result in recognition accuracy drop [20]. In our work, we explore the possibility of face recognition without alignment. In the proposed algorithm, we concentrate on separating factors that might affect recognition accuracy. To be specific, we use bilinear model to separate a varying factor from recognition target. And we prove in experiments that this separation is effective and required.

Bilinear models [8] are usually used to model systems with two factors and each factor itself is linear given the other factor fixed. Bilinear models for 2D image data have been widely used to solve face recognition problems due to their simplicity in formulation. In this category of solutions, factors introduced in a system are modeled as a symmetric bilinear model or asymmetric bilinear model. In this work, we also introduce asymmetric bilinear model due to its simplicity in representation.

Among all variance factors in face recognition problems, illumination change is one influential and intensively studied factor. Example works on illumination variations in face recognition problem include [9, 10], of which work in [10] is closely related to our work. Authors in [10] combine ridge regression into symmetric bilinear model to separate illumination and identity factor. Symmetric bilinear model decomposes input data into two factor matrices representing two factors and an interaction matrix denoting interactions between two factors. Instead of symmetric models, we utilize asymmetric models which decompose input data into a factor matrix and a matrix containing the other factor and interaction matrix. With fewer parameters to optimize, the asymmetric model does not suffer from convergence problem as in symmetric model. But, due to noise captured in image features, the original asymmetric model is not able to cope with illumination changes in real world face recognition problems.

In this work, we propose a modified asymmetric model for accurate face recognition. As in the original asymmetric bilinear model, we calculate probability predictions with Bayesian rules and expectation maximization method (EM). But instead of initializing the factor probabilities randomly, we initialize them with nearest neighbor method and optimize them for the test data. And we update the factor model to be identified. The enhanced asymmetric model is composed of three modules: calculating factor matrices from training data, initialization step for test data, and optimization and classification step for test data. In the first step, we use singular value decomposition to separate the factor to be identified from other factors based on training data. In the initialization step, a nearest neighbor method is introduced to initialize the joint probabilities of all factors which is further updated in optimization step. In the third step, we update the factor model to be identified. We validate the proposed method on a manually created data and cropped images from extended Yale B dataset. The experiment results show that the enhanced asymmetric models give promising results and good recognition accuracies.

The rest of this paper is organized as follows: in Section 2 we explain the proposed method in detail, including how to use Bayesian rules and EM method to optimize test factors and how to achieve illumination-robust face recognition and identity-robust illumination recognition goal with the proposed method; in Section 3, we validate the proposed method on two types of data: designed data and public available extended Yale B dataset; in Section 4, we will conclude our work and discuss possible future works.

2. Enhanced Asymmetric Bilinear Model

Bilinear models [8] separate the input data into two factors, described as style and content. Content is the factor to be identified; for example, identity and style are the factors that vary and need to be separated. Bilinear models reduce to the linear model when one factor is fixed. It contains symmetric and asymmetric models. Authors in [10] use symmetric model and take the identity as content and the illumination as style. In a symmetric bilinear model, style and content interact using an interaction matrix. But actually we are not interested in both style and content matrices in a single task classification problem. For simplicity, we choose an asymmetric model and modify it to accurately recognize factors.

Suppose we represent content c with a vector of parameters ${\mathbf{b}}^c$ of dimension J and use ${\mathbf{y}}^{sc}$ to denote K dimensional input feature vector of style s and content c; an asymmetric bilinear model represents the observation vector y as

$$\begin{array}{cc}& \hspace{0.17em}{y}_k^{sc}\hspace{0.17em}={\displaystyle \sum }_j‍a_{jk}^s{b}_j^c.\end{array}$$

(1)

This formulation can also be written as

$$\begin{array}{cc}& {\mathbf{y}}^{sc}={\mathbf{A}}^s{\mathbf{b}}^c,\end{array}$$

(2)

where ${\mathbf{A}}^s$ denotes $K\times J$ matrix with entries $\{a_{jk}^s\}$ and

$$\begin{array}{cc}& a_{jk}^s={\displaystyle \sum }_i‍{\omega }_{ijk}^s{a}_i^s.\end{array}$$

(3)

In a symmetric model, ω and a in (3) are represented separately thus resulting in one more variable.

As in [8], we use singular value decomposition to fit asymmetric model to training data. First we stack $SC\times K$ -dimensional input features as follows:

$$\begin{array}{cc}& \stackrel{-}{\mathbf{Y}}=\left[\begin{array}{ccc}{\stackrel{-}{\mathbf{y}}}^{\mathrm{11}}& \cdots & {\stackrel{-}{\mathbf{y}}}^{\mathrm{1}C}\\ ⋮& \ddots & ⋮\\ {\stackrel{-}{\mathbf{y}}}^{S\mathrm{1}}& \cdots & {\stackrel{-}{\mathbf{y}}}^{SC}\end{array}\right];\end{array}$$

(4)

then we apply singular value decomposition to the observation:

$$\begin{array}{cc}& \stackrel{-}{\mathbf{Y}}=\mathbf{U}\mathbf{S}{\mathbf{V}}^T.\end{array}$$

(5)

The style matrix A is initialized as the first J columns of $\mathbf{U}\mathbf{S}$ and the content matrix B is initialized as the first J rows of ${\mathbf{V}}^T$ .

2.1. Optimizing Factor Matrix for Test Data

Given test data $\tilde{\mathbf{Y}}$ of an unseen style $\tilde{s}$ , the goal is to find the matrix containing style factor ${\mathbf{A}}^{\tilde{s}}$ and content factor ${\mathbf{b}}^{\tilde{c}}$ approximating test data as precious as possible, that is, to minimize the approximation error E,

$$\begin{array}{cc}& E=‖\tilde{\mathbf{Y}}-{\mathbf{A}}^{\tilde{s}}{\mathbf{b}}^{\tilde{c}}‖.\end{array}$$

(6)

In original bilinear model, optimization is under several assumptions: style matrix of the test data ${\mathbf{b}}^{\tilde{c}}$ equals style matrix of the training data ${\mathbf{b}}^{\tilde{c}}$ ; the probability of test observation of a specific style and a specific content does not need special way of initialization. Due to captured noise in input feature, the problem in reality is not preciously a linear model. So we add initialization step and update content matrix in optimization. According to experimental results, updating the content matrix ${\mathbf{b}}^{\tilde{c}}$ instead of keeping it unchanged gives more accurate recognition results.

2.1.1. Bayesian Rules for Probability Predictions

We define the probability of a captured feature given a style and a content assignment $p(\tilde{\mathbf{y}}\hspace{0.17em}|\hspace{0.17em}\tilde{s},\tilde{c})$ as a Gaussian distribution; that is,

$$\begin{array}{cc}& p(\tilde{\mathbf{y}}\hspace{0.17em}|\hspace{0.17em}\tilde{s},\tilde{c})=\exp \{-‖\tilde{\mathbf{y}}-{\mathbf{A}}^{\tilde{s}}{\mathbf{b}}^{\tilde{c}}‖\}.\end{array}$$

(7)

And we use Bayesian rules to calculate probabilities $p(\tilde{s},\tilde{c}\hspace{0.17em}|\hspace{0.17em}\tilde{\mathbf{y}})$ , which is later on utilized to update matrix ${\mathbf{A}}^{\tilde{s}}$ :

$$\begin{array}{cc}& p(\tilde{s},\tilde{c}\hspace{0.17em}|\hspace{0.17em}\tilde{\mathbf{y}})=\frac{p(\tilde{\mathbf{y}}\hspace{0.17em}|\hspace{0.17em}\tilde{s},\tilde{c})p\left(\tilde{\mathbf{y}}\hspace{0.17em}|\hspace{0.17em}\tilde{s}\right)}{\tilde{\mathbf{y}}}.\end{array}$$

(8)

This variable is introduced as a weight in optimization step. Instead of the random initialization, we set the initial $p(\tilde{\mathbf{y}}\hspace{0.17em}|\hspace{0.17em}\tilde{s},\tilde{c})$ as $\mathrm{1}$ if $\tilde{\mathbf{y}}$ has a nearest neighbor ${\mathbf{y}}_n$ with a style $\tilde{s}$ and a content $\tilde{c}$ and $\mathrm{0}$ otherwise.

2.1.2. The Algorithm

To minimize the error between the test features and its approximation ${\mathbf{A}}^{\tilde{s}}{\mathbf{b}}^{\tilde{c}}$ , we use an EM and add an extra update of the matrix containing content ${\mathbf{b}}^{\tilde{c}}$ which proves to give more accurate recognition results.

• (1)

Initialize the matrix containing style factors $A^{\tilde{s}}$ as being equal to that computed from training data A, and initialize $p(\tilde{\mathbf{y}}\hspace{0.17em}|\hspace{0.17em}\tilde{s},\tilde{\text{c}})$ as $\mathrm{1}$ if $\tilde{\mathbf{y}}$ has a nearest neighbor ${\mathbf{y}}_n$ with a style $\tilde{s}$ and a content $\tilde{c}$ , and otherwise 0.

(2)

Expectation step: calculate $p(\tilde{s},\tilde{c}\hspace{0.17em}|\hspace{0.17em}\tilde{\mathbf{y}})$ with (8).

(3)

Maximization step: to maximize the total log-likelihood of the test data $L^{*}={\sum }_y‍\log p(\mathbf{y})$ , calculate the gradient of $L^{*}$ with respect to the matrix containing style, that is, solving $\partial L^{*}/\partial {\mathbf{A}}^{\tilde{\mathbf{s}}}=\mathrm{0}$ . And we get ${\mathbf{A}}^{\tilde{\mathbf{s}}}=[{\sum }_c‍{\mathbf{m}}^{\tilde{s}\widetilde{\hspace{0.17em}c}}{{\mathbf{b}}^{\tilde{c}}}^T][{\sum }_c‍n^{\tilde{s}\hspace{0.17em}\tilde{c}}{\mathbf{b}}^{\tilde{c}}{{\mathbf{b}}^{\tilde{c}}}^T{]}^{-\mathrm{1}}$ , where ${\mathbf{m}}^{\tilde{s}\widetilde{\hspace{0.17em}c}}={\sum }_{\mathbf{y}}‍p(\tilde{s},\tilde{c}\hspace{0.17em}|\hspace{0.17em}\tilde{\mathbf{y}})\mathbf{y}$ and $n^{\widetilde{s\hspace{0.17em}}\tilde{c}}={\sum }_{\mathbf{y}}‍p(\tilde{s},\tilde{c}\hspace{0.17em}|\hspace{0.17em}\tilde{\mathbf{y}})$ .

(4)

Update content matrix $\tilde{\mathbf{B}}=({\mathbf{Y}}^T{\mathbf{A}}^T)/(\mathbf{A}{\mathbf{A}}^T)$ .

2.1.3. Illumination-Robust Face Recognition and Identity-Robust Illumination Recognition

In this section, we explain the possible applications of the proposed method in face recognition problems. First, we define identity as content and illumination as style to show the application of linear models to illumination-robust face recognition problem; see Figure 1. Then we switch content and style and set the illumination as content and identity as style. It is widely acknowledged that the illumination changes could be modeled with linear space but it is not clear if the identity space is also the same case. We design the second experiment to show the application of the proposed method in identity-robust illumination recognition and to test the expansibility of the system. In the following section, we show experimental results of these two experiment settings.

Figure 1

In the illumination-robust identity recognition application, we show the split between the training and test data.

3. Experimental Results

We validate the proposed method with two categories of data: one is a manually defined sample data and another is a publicly available face recognition dataset. The aim of the designed sample data is to verify the soundness of the proposed method and the second category of experiments is to provide standard results.

3.1. Experiment on Designed Data

Suppose we have a tensor $\mathcal{A}$ of size $\mathrm{4}\times \mathrm{4}\times \mathrm{11}$ with the first dimension representing content, the second style, and the third feature dimensions. Entries of the tensor are defined as

$$\begin{array}{cc}& \mathcal{Y}\left(\mathrm{1},:,:\right)=\left(\begin{array}{ccccccccccc}1& 0& 0& 0& 0& 0& 0& 0& 0& 0& 0\\ 1& 0& 1& 0& 0& 0& 0& 0& 0& 0& 0\\ 1& 0& 1& 0& 1& 0& 0& 0& 0& 0& 0\\ 1& 0& 1& 0& 1& 0& 1& 0& 0& 0& 0\end{array}\right);\\ \mathcal{Y}\left(\mathrm{2},:,:\right)=\left(\begin{array}{ccccccccccc}0& 1& 0& 0& 0& 0& 0& 0& 0& 0& 0\\ 0& 1& 0& 1& 0& 0& 0& 0& 0& 0& 0\\ 0& 1& 0& 1& 0& 1& 0& 0& 0& 0& 0\\ 0& 1& 0& 1& 0& 1& 0& 1& 0& 0& 0\end{array}\right);\\ \mathcal{Y}\left(\mathrm{3},:,:\right)=\left(\begin{array}{ccccccccccc}1& 0& 0& 0& 0& 0& 0& 0& 0& 0& 0\\ 1& 0& 0& 1& 0& 0& 0& 0& 0& 0& 0\\ 1& 0& 0& 1& 0& 0& 1& 0& 0& 0& 0\\ 1& 0& 0& 1& 0& 0& 1& 0& 0& 1& 0\end{array}\right);\end{array}$$

(9)

here content denotes the total number of ones in a feature vector and style denotes which positions are set as ones. For example, in the first style, ones occur in the first, third, fifth, and seventh columns. If we use all contents and styles for training and select one content to test the method, the recognition accuracies should reach $\mathrm{100}$ percent. So we choose the whole tensor as training data and pick $\tilde{\mathbf{Y}}$ as $\mathcal{Y}(\mathrm{2},:,:)$ .

With the proposed method, we get the experimental results in Table 1. The tables show recognition accuracies and approximation errors in each iteration. From the table, we can see that the proposed method can reach 100 percent recognition accuracy on the designed dataset.

Table 1

Recognition accuracies and approximation errors of the enhanced asymmetric model on the designed data sample.

Iteration number	1	2	3	4	5	6	7	8	9
Accuracies	0.25	1.00	1.00	1.00	1.00	1.00	1.00	1.00	0.75
Error	100000	5.20	5.62	5.78	6.17	3.33	3.66	5.18	7.16

3.2. Experiment on Extended Yale B Dataset

The extended Yale Face Database B contains 16128 images of 28 human subjects under 9 poses and 64 illumination conditions and they provide a subset of cropped images. In our experiment we use the cropped images. With this dataset, we design two sets of experiments: illumination-robust face recognition and identity-robust illumination recognition.

In illumination-robust face recognition, we pick those illuminations with id ranging from $\mathrm{1}$ to 39 as training and illuminations with id 50 for test. The experimental results for this experiment setting are shown in Table 2. From the table, we can see that after $\mathrm{3}$ iterations the recognition accuracy reaches the maximum 0.92 and then starts decreasing dramatically.

Table 2

Recognition accuracies and approximation errors of the enhanced asymmetric model on illumination-robust face recognition experiment.

Iteration number	1	2	3	4	5	6
Accuracies	0.13	0.56	0.92	0.92	0.74	0.44
Error (divided by $$\mathit{1.0}e+\mathit{12}$$ )	0.38	0.0031	0.0023	0.0024	0.0024	0.0025

In identity-robust illumination recognition, we pick those persons with id ranging from $\mathrm{1}$ to 38 as training and person with id 39 as test. The experimental results for this experiment setting are shown in Table 3. From the table, we can see that although with similar amount of training data, illumination recognition gives better results than identity recognition. This might be due to sufficient training data (38) for the identity space or illumination to be an easier target. From Tables 2 and 3, we can say that the enhanced asymmetric model gives high recognition accuracies. Recognition accuracies increase quite fast with approximation error decreasing and with several steps, we can achieve the optimum recognition accuracies for our method.

Table 3

Recognition accuracies and approximation errors of the enhanced asymmetric model on identity-robust illumination recognition experiment.

Iteration number	1	2	3	4
Accuracies	0.88	1.0000	0.9219	0.3906
Error (divided by $$\mathit{1.0}e+\mathit{11}$$ )	5.5517	0.1521	0.1679	0.2006

We further show a set of experiments exploring the effect of training sample numbers. In illumination-robust identity recognition application, we pick the same test data as the previous experiment but we pick the illumination with identity ranging from $\mathrm{1}$ to $\mathrm{5}$ as training in experiment $\mathrm{1}$ (Exp1), illuminations with identity ranging from $\mathrm{1}$ to $\mathrm{9}$ as training in experiment $\mathrm{2}$ (Exp2), and illuminations with identity ranging from $\mathrm{1}$ to $\mathrm{13}$ as training in experiment $\mathrm{3}$ (Exp3). Experimental results are shown in Figure 2.

Figure 2

Experimental results of different training/test splits for illumination-robust identity recognition applications.

From the figure, we can see that, in the term of errors, the training sample numbers differences have an obvious effect in the first and the second iterations but tend to be subtle in the following iterations. In fact, with more training sample numbers, the optimization tends to be unstable. The best accuracy $\mathrm{92.31}\%$ occurs in all experiment settings. In the third iteration, experiment $\mathrm{1}$ and experiment $\mathrm{2}$ reach the maximum while experiment $\mathrm{3}$ reaches the maximum in iteration $\mathrm{6}$ .

The proposed algorithm is further compared with a nearest neighbor method to show the importance of separating style and content. We design the nearest neighbor method applied in identity recognition problem: first, varying numbers of training data are selected (i.e., $\mathrm{5}$ , $\mathrm{9}$ , and $\mathrm{13}$ ); then, for each test data, it is compared with all styles (illumination variations in this case) from all contents (face identity in this case); finally, the label of the test data is set as the label of the training data with the minimum distance. The recognition accuracy of the nearest neighbor method is shown in Table 4. From the table, we can see that the raw nearest neighbor method performs very badly on the identity recognition problem here. What is more, when the number of training data increases, the recognition accuracy drops. Note that each training data case has a different style compared to other training data in the dataset. We can see from the recognition accuracies that more training data from variant styles confuses the method. Adding different styles of training data without separating them makes the recognition accuracies worse.

Table 4

Recognition accuracies of the nearest neighbor method on identity recognition experiment.

Training case number	5	9	13
Accuracies	0.0526	0.0526	0.0263

4. Conclusions and Future Works

In this paper, we present an enhanced asymmetric bilinear model and apply it to illumination-robust face recognition problem and identity-robust illumination recognition problems. By initializing the content and style probability with nearest neighbor method and adding update for content matrix, we achieve quite good recognition accuracies. In the future, it would be interesting to explore the application of the proposed method on models containing more than two factors. Also, we would like to explore applications of asymmetric models on applications other than face recognition.