Multiple Kernel Learning in Fisher Discriminant Analysis for Face Recognition

Abstract

Recent applications and developments based on support vector machines (SVMs) have shown that using multiple kernels instead of a single one can enhance classifier performance. However, there are few reports on performance of the kernel-based Fisher discriminant analysis (kernel-based FDA) method with multiple kernels. This paper proposes a multiple kernel construction method for kernel-based FDA. The constructed kernel is a linear combination of several base kernels with a constraint on their weights. By maximizing the margin maximization criterion (MMC), we present an iterative scheme for weight optimization. The experiments on the FERET and CMU PIE face databases show that, our multiple kernel Fisher discriminant analysis (MKFD) achieves high recognition performance, compared with single-kernel-based FDA. The experiments also show that the constructed kernel relaxes parameter selection for kernel-based FDA to some extent.

1. Introduction

As there exist many image variations such as pose, illumination and facial expression, face recognition is a highly complex and nonlinear problem which could not be sufficiently handled by linear methods, such as principal components analysis (PCA) [1] and Fisher discriminant analysis (FDA) [2]. Therefore, it is reasonable to assume that a better solution to this inherent nonlinear problem could be achieved using nonlinear methods, such as the so-called kernel machine techniques [3]. Following the success of applying the kernel trick in SVMs, many kernel-based PCA and FDA methods have been developed and applied in pattern recognition tasks, such as kernel PCA (KPCA) [4], kernel Fisher discriminant (KFD) [5], generalized discriminant analysis (GDA) [6], and kernel direct FDA (KDDA) [7].

It has been shown that the kernel-based FDA method is a feasible approach to solve the nonlinear problems in face recognition. However, the performance of the kernel-based FDA method is sensitive to the selection of a kernel function and its parameters. Kernel parameter selection to date can mainly be achieved by Cross Validation [8], which is computationally expensive, and the selected kernel parameters can not be guaranteed optimal. Furthermore, a single and fixed kernel can only characterize the geometrical structure of some aspects for the input data and, thus, not always be fit for the applications which involve data from multiple, heterogeneous sources [9][10], such as face images under broad variations of pose, illumination, facial expression, aging, etc.

Recent applications and developments based on SVMs [11][12] have shown that using multiple kernels (i.e., a combination of several “base kernels”) instead of a single fixed one can enhance classifier performance, which raised the so-called multiple kernel learning (MKL) method. With m kernels, input data can be mapped into m feature spaces, where each feature space can be taken as one view of the original input data [10]. Each view is expected to exhibit some geometrical structures of the original data from its own perspective such that all the views can complement for the subsequent learning task. It has been proven that MKL can offer some needed flexibility and well manipulate the case that involves multiple, heterogeneous data sources [9][13][14]. However, MKL is proposed for SVMs, and there have been few reports on performance of the kernel-based FDA method with multiple kernels.

In this paper, we propose multiple kernel Fisher discriminant analysis (MKFD), in which the constructed kernel is a linear combination of several base kernels with a constraint on their weights, and we give an iterative scheme for weight optimization.

The rest of this paper is organized as follows. First we describe the kernel construction for MKFD in section 2. Then in section 3, the optimization scheme for the multi-kernel weights is presented. The experimental results are reported in section 4, while we draw our conclusion in section 5.

2. Kernel construction for MKFD

Given M Mercer kernel functions k^(m)(x, y), m = 1, 2,…,M, defined on ℝ ^d × ℝ ^d , which are M base kernels, we construct the multiple kernel function as the following linear combination

$k(x,y)={\displaystyle \sum _{m=1}^M{\gamma }_m{}^2{k}^{(m)}(x,y)},\mathrm{s}.\mathrm{t}.{\displaystyle \sum _{m=1}^M{\gamma }_m}=1,$

(1)

where γ² _m is the weight of base k^(m) (x, y). Since the weights are nonnegative, it is easy to show k (x, y) is also a Mercer kernel defined on ℝ ^d × ℝ ^d .

We apply the multiple kernel function (1) in kernel-based FDA, which produces what we call MKFD. To achieve good performance of MKFD for face recognition, we consider the problem of learning proper weights of the base kernels, i.e., choosing the best γ = {γ₁, γ₂,…,γ _M )^T, without regard to the specific structures of base kernel functions.

3. Weight optimization for MKFD

3.1 Some notations on MKFD

Let 𝕏 ⊂ ℝ ^d be the original sample space. Let X be a training set of N samples and C be the number of sample classes. Assume the i th class X_i contains N_i samples, i.e., $X_i=\{x_1^i,x_2^i,\dots ,x_{N_i}^i\}$ , $i=1,2,\dots ,C$ so $N={\displaystyle {\sum }_{i=1}^C{N}_i}$ .

Denote M nonlinear mappings as ${\mathrm{\Phi }}_m:x\in \mathbb{X}\to {\mathrm{\Phi }}_m(x)\in \mathbb{F}$ , $m=1,2,\mathrm{...},M$ , where F is the mapped feature space, with df = dim F (dimensionality of F). Denote M base kernel matrices (N × N) as

$K^{(m)}={\left[k^{(m)}(x_j^i,x_h^l)\right]}_{\begin{array}{l}i=1,\dots ,C,j=1,\dots ,N_i\\ l=1,\dots ,C,h=1,\dots ,N_l\end{array}},m=1,\mathrm{...},M,$

(2)

where $k^{(m)}(x_j^i,x_h^l)={\mathrm{\Phi }}_m{(x_j^i)}^{\mathrm{T}}{\mathrm{\Phi }}_m(x_h^l)$ , each base kernel corresponding to one nonlinear mapping.

Given γ = (γ₁, γ₂,…,γ) subject to σ ^M _m=1γm = 1, N × N multiple kernel matrix is

$K={\displaystyle \sum _{m=1}^M{\gamma }_m{}^2{K}^{(m)}}.$

(3)

We call the mapping from 𝕏 to F multiple nonlinear mapping, denoted as φ, which is implicitly defined by the multiple kernel matrix K, and can be understood as the compound of φ _m , m = 1,…,M.

Under multiple nonlinear mapping φ, the i th mapped class and the mapped sample set are respectively given by

$\begin{array}{l}\mathrm{\Phi }(X_i)=\{\mathrm{\Phi }(x_1^i),\mathrm{\Phi }(x_2^i),\dots ,\mathrm{\Phi }(x_{N_i}^i)\},\\ \mathrm{\Phi }(X)=\{\mathrm{\Phi }(X_1),\mathrm{\Phi }(X_2),\dots ,\mathrm{\Phi }(X_C)\}.\end{array}$

Also, the mean of the mapped class φ(X_i) and that of the mapped sample set φ(X) are respectively given by

$m_i=\frac{1}{N_i}{\displaystyle \sum _{j=1}^{N_i}\mathrm{\Phi }(x_j^i)},m=\frac{1}{N}{\displaystyle \sum _{i=1}^C{\displaystyle \sum _{j=1}^{N_i}\mathrm{\Phi }(x_j^i)}}.$

In kernel feature space 𝕏, the within-class scatter matrix S^Φ _w and between-class scatter matrix S^Φ _b are respectively defined as

$S_w^{\mathrm{\Phi }}=\frac{1}{N}{\displaystyle \sum _{i=1}^C{\displaystyle \sum _{x\in X_i}(\mathrm{\Phi }(x)-m_i){(\mathrm{\Phi }(x)-m_i)}^{\mathrm{T}}}}={\mathrm{\Phi }}_w{\mathrm{\Phi }}_w^{\mathrm{T}},$

(4)

$S_b^{\mathrm{\Phi }}=\frac{1}{N}{\displaystyle \sum _{i=1}^C{N}_i(m_i-m){(m_i-m)}^{\mathrm{T}}}={\mathrm{\Phi }}_b{\mathrm{\Phi }}_b^{\mathrm{T}},$

(5)

Where

$\begin{array}{c}{\mathrm{\Phi }}_w={\left[{\phi }_1^1,\mathrm{...},{\phi }_{N_1}^1,{\phi }_1^2,\mathrm{...},{\phi }_{N_2}^2,\mathrm{......},{\phi }_1^C,\mathrm{...},{\phi }_{N_C}^C\right]}_{df\times N},\\ {\phi }_j^i=\frac{1}{\sqrt{N}}\left(\mathrm{\Phi }(x_j^i)-m_i\right),\\ {\mathrm{\Phi }}_b={\left[{\phi }_1,\mathrm{...},{\phi }_C\right]}_{df\times C},{\phi }_i=\sqrt{\frac{N_i}{N}}(m_i-m).\end{array}$

(6)

The kernel Fisher criterion is defined as

$J^{\mathrm{\Phi }}(W)=\frac{\text{tr}(W^{\mathrm{T}}{S}_b^{\mathrm{\Phi }}W)}{\text{tr}(W^{\mathrm{T}}{S}_w^{\mathrm{\Phi }}W)},$

where W = {w_1,…,w _q } is a df × q (df > q) projection matrix. MKFD is to find an optimal projection matrix W* : ℝ ^df → ℝ ^q in mapped feature space F, such that W* = arg max J ^φ (W).

3.2 Diagonalization strategy

We use the same diagonalization strategy as KDDA [7] to deal with the small sample size (SSS) problem in our MKFD, i.e., first diagonalzing S^Φ _b to I (identical matrix) and then diagonalzing S^Φ _w to Λ _w , which is briefly expressed using the MKFD notations as follows.

3.2.1 Eigen-analysis of S^Φ _w in the feature space.

Φ^T _b can be expressed using the multiple kernel matrix K as follows:

$\begin{array}{c}{\mathrm{\Phi }}_b^{\mathrm{T}}{\mathrm{\Phi }}_b=\frac{1}{N}D\cdot (A_{NC}^{\mathrm{T}}\cdot K\cdot A_{NC}-\frac{1}{N}{A}_{NC}^{\mathrm{T}}\cdot K\cdot 1_{NC}\\ \hspace{1em}\hspace{1em}\hspace{1em}-\frac{1}{N}{1}_{NC}^{\mathrm{T}}\cdot K\cdot A_{NC}+\frac{1}{N^2}{1}_{NC}^{\mathrm{T}}\cdot K\cdot 1_{NC})\cdot D,\end{array}$

(7)

where $D=\text{diag}(\sqrt{N_1},\mathrm{...},\sqrt{N_C})$ (C × C diagnal matrix), 1 _NC is a N × C matrix with terms all equal to one, A _NC = diag (a_N1,…, a _Nc ) is a N × C block diagonal matrix, and aN_i is a N_i × 1 vector with all terms equal to $\frac{1}{N_i}$ .

Let λ _i and e _i (i = 1,…,C) be the i th largest eigenvalue and corresponding eigenvector of Φ^T _b ṁ Let r (≤ C – 1) be the rank of S^Φ_b(= Φ_bΦ^T_b) (also the rank of Φ_bΦ^T_b). Denote E _r = (e_1,…,e _r ), and V = (v_1,…, v _r ) = φ _b E _r . It can be derived that V^TSΦV = Φ _b , with Λ _b =diag(γ²₁,…,γ²_r), a nonsingular diagonal matrix. Let U = VΦ _b . Then U^TS^φ _b U = 1.

3.2.2 Eigen-analysis of S^Φ _w in the feature space

Based on the analysis in section 3.2.1, it can be seen that

$U^{\mathrm{T}}{S}_w^{\mathrm{\Phi }}U={(E_r{\mathrm{\Lambda }}_b^{-1/2})}^{\mathrm{T}}({\mathrm{\Phi }}_b^{\mathrm{T}}{S}_w^{\mathrm{\Phi }}{\mathrm{\Phi }}_b)(E_r{\mathrm{\Lambda }}_b^{-1/2}),$

where ${\mathrm{\Phi }}_b^{\mathrm{T}}{S}_w^{\mathrm{\Phi }}{\mathrm{\Phi }}_b$ can be expressed using K, with the similar details seen in [7].

Let z _j be the eigenvector of U^TSΦU corresponding to the j th smallest eigenvalue λ' _j , j = 1,…,r. Denote Z = (z₁,…,z _r ). Defining Y = UZ, it can be derived that Y^TS^Φ _w Y = Λ _w , with Λ _w = diag(λ₁',…,λ _r ’).

Based on the derivation presented in section 3.2.1 and 3.2.2, an optimal projection matrix for MKFD is obtained as

$W*=Y{\mathrm{\Lambda }}_w^{-1/2}={\mathrm{\Phi }}_b{E}_r{\mathrm{\Lambda }}_b^{-1/2}Z{\mathrm{\Lambda }}_w^{-1/2}.$

(8)

Certainly, as the multiple nonlinear mapping φ is implicitly defined by the multiple kernel function (or matrix), φ _b (defined by Eq. (6)) remains unknown, and W* can not be evaluated. The real meaning of Eq. (8) is obtaining matrix $E_r{\Lambda }_b^{-1/2}Z{\mathrm{\Lambda }}_w^{-1/2}$ , which can be computed from the multiple kernel matrix K. This is the core result of diagonalization for MKFD.

3.3 Optimization criterion and objective

We adopt the maximum margin criterion (MMC) [15] as the objective function to optimize weight γ:

$F(W,\gamma )=\text{tr}(W^{\mathrm{T}}{S}_b^{\mathrm{\Phi }}W)-\text{tr}(W^{\mathrm{T}}{S}_w^{\mathrm{\Phi }}W),$

(9)

where W is a projection matrix, γ = {γ₁, γ₂,…,γ _M )^T, subject to 1^T γ = ∊γ _m = 1, with γ² _m being the weight of the m th base kernel matrix K^(m).

Based on the result (8) in 3.2, the optimal projection matrix $W*={\mathrm{\Phi }}_b{E}_r{\mathrm{\Lambda }}_b^{-1/2}Z{\mathrm{\Lambda }}_w^{-1/2}$ . Denoting $G=E_r{\mathrm{\Lambda }}_b^{-1/2}Z{\mathrm{\Lambda }}_w^{-1/2}$ , which can be computed from the multiple kernel matrix K. Then the objective function (9) can be reformulated as

$\begin{array}{l}F(\gamma )=\text{tr}(W{*}^{\mathrm{T}}{S}_b^{\mathrm{\Phi }}W*-W{*}^{\mathrm{T}}{S}_w^{\mathrm{\Phi }}W*)\\ =\text{tr}(G^{\mathrm{T}}{\mathrm{\Phi }}_b^{\mathrm{T}}{\mathrm{\Phi }}_b{\mathrm{\Phi }}_b^{\mathrm{T}}{\mathrm{\Phi }}_{b}G-G^{\mathrm{T}}{\mathrm{\Phi }}_b^{\mathrm{T}}{\mathrm{\Phi }}_w{\mathrm{\Phi }}_w^{\mathrm{T}}{\mathrm{\Phi }}_{b}G)\\ =\text{tr}(G^{\mathrm{T}}P{P}^{\mathrm{T}}G-G^{\mathrm{T}}Q{Q}^{\mathrm{T}}G),\end{array}$

(10)

where P = Φ^T _b Φ _b and Q = Φ^T _b Φ _w can be expressed in terms of the multiple kernel matrix K as follows.

$\begin{array}{l}P=\frac{1}{N}D\cdot (A_{NC}^{\mathrm{T}}\cdot K\cdot A_{NC}-\frac{1}{N}{A}_{NC}^{\mathrm{T}}\cdot K\cdot 1_{NC}\\ \hspace{0.17em}\hspace{1em}-\frac{1}{N}{1}_{NC}^{\mathrm{T}}\cdot K\cdot A_{NC}+\frac{1}{N^2}{1}_{NC}^{\mathrm{T}}\cdot K\cdot 1_{NC})\cdot D\\ \hspace{1em}={\displaystyle \sum _{m=1}^M{\gamma }_m{}^2{P}^{(m)},}\end{array}$

(11)

where

$\begin{array}{c}Q=\frac{1}{N}D\cdot (A_{NC}^{\mathrm{T}}\cdot K-A_{NC}^{\mathrm{T}}\cdot K\cdot H_{NN}\\ -\frac{1}{N}{1}_{NC}^{\mathrm{T}}\cdot K+\frac{1}{N}{1}_{NC}^{\mathrm{T}}\cdot K\cdot H_{NN})\\ ={\displaystyle \sum _{m=1}^M{\gamma }_m{}^{ 2}{Q}^{(m)}},\end{array}$

with D, A _NC and 1 _NC defined the same as in (7);

$\begin{array}{l}Q=\frac{1}{N}D\cdot (A_{NC}^{\mathrm{T}}\cdot K-A_{NC}^{\mathrm{T}}\cdot K\cdot H_{NN}\\ \hspace{1em}\hspace{1em}-\frac{1}{N}{1}_{NC}^{\mathrm{T}}\cdot K+\frac{1}{N}{1}_{NC}^{\mathrm{T}}\cdot K\cdot H_{NN})\\ \hspace{1em}={\displaystyle \sum _{m=1}^M{\gamma }_m{}^2{Q}^{(m)}},\end{array}$

(12)

where

$\begin{array}{c}{Q}^{(m)}=\frac{1}{N}D\cdot (A_{NC}^{\mathrm{T}}\cdot K^{(m)}-A_{NC}^{\mathrm{T}}\cdot K^{(m)}\cdot H_{NN}\\ -\frac{1}{N}{1}_{NC}^{\mathrm{T}}\cdot K^{(m)}+\frac{1}{N}{1}_{NC}^{\mathrm{T}}\cdot K^{(m)}\cdot H_{NN}),\\ m=1,2,\mathrm{...},M,\end{array}$

H _NN = diag(h_N1,…,h _NC ) is a N × N block diagonal matrix, and h _Ni is a N_i × N_i matrix with all terms equal to $\frac{1}{N_i}$ .

Therefore, to find the best weights (γ₁, γ₂, …, γ _M ) for the multiple kernel matrix K defined in Eq. (3), we need solve the following constrained optimization

$\begin{array}{l}\underset{\gamma }{\max }\hspace{1em}F(\gamma )=\text{tr}(G^{\mathrm{T}}P{P}^{\mathrm{T}}G-G^{\mathrm{T}}Q{Q}^{\mathrm{T}}G)\\ \hspace{1em}\mathrm{s}.\mathrm{t}.\hspace{1em}{1}^{\mathrm{T}}\gamma ={\displaystyle \sum _{m=1}^M{\gamma }_m}=1.\end{array}$

(13)

3.4 Solving the optimization problem

We introduce a Lagrangian

$\mathbb{L}(\gamma ,\alpha )=F(\gamma )+\alpha ({\displaystyle \sum _{m=1}^M{\gamma }_m}-1),$

(14)

with one multiplier α. From Eq. (11) and (12), we can obtain

$\frac{\partial P}{\partial {\gamma }_m}=2{\gamma }_m{P}^{(m)},\frac{\partial Q}{\partial {\gamma }_m}=2{\gamma }_m{Q}^{(m)}.$

Moreover, temporarily regarding G as constant, we have

$\begin{array}{l}\frac{\partial F(\gamma )}{\partial {\gamma }_m}=\frac{\partial }{\partial {\gamma }_m}(\text{tr}(G^{\mathrm{T}}P{P}^{\mathrm{T}}G-G^{\mathrm{T}}Q{Q}^{\mathrm{T}}G))\\ =\text{tr}(\frac{\partial }{\partial {\gamma }_m}(G^{\mathrm{T}}P{P}^{\mathrm{T}}G-G^{\mathrm{T}}Q{Q}^{\mathrm{T}}G))\\ =\text{tr}((G^{\mathrm{T}}\frac{\partial P}{\partial {\gamma }_m}{P}^{\mathrm{T}}G+G^{\mathrm{T}}P\frac{\partial P^{\mathrm{T}}}{\partial {\gamma }_m}G)\\ \hspace{1em}-(G^{\mathrm{T}}\frac{\partial Q}{\partial {\gamma }_m}{Q}^{\mathrm{T}}G+G^{\mathrm{T}}Q\frac{\partial Q^{\mathrm{T}}}{\partial {\gamma }_m}G))\\ =2{\gamma }_m{\displaystyle \sum _{k=1}^M{\gamma }_k{}^2{\text{tr}}_{m,k}},\end{array}$

(15)

Where

$\begin{array}{c}{\text{tr}}_{m,k}=\text{tr}((G^{\mathrm{T}}{P}^{(m)}{P}^{(k)}{}^{\mathrm{T}}G+G^{\mathrm{T}}{P}^{(k)}{P}^{(m)}{}^{\mathrm{T}}G)\\ -(G^{\mathrm{T}}{Q}^{(m)}{Q}^{(k)}{}^{\mathrm{T}}G+G^{\mathrm{T}}{Q}^{(k)}{Q}^{(m)}{}^{\mathrm{T}}G)),\\ m,k=1,2,\mathrm{...},M.\end{array}$

Now, differentiating L(γ, α) with respect to γ₁,…, γ _M and α gives the following partial derivatives:

$\begin{array}{r}\frac{\partial \mathbb{L}(\gamma ,\alpha )}{\partial {\gamma }_m}=2{\gamma }_m{\displaystyle \sum _{k=1}^M{\gamma }_k{}^2{\text{tr}}_{m,k}}+\alpha \\ m=1,\mathrm{...},M,\end{array}$

(16)

$\frac{\partial \mathbb{L}(\gamma ,\alpha )}{\partial \alpha }={\displaystyle \sum _{m=1}^M{\gamma }_m}-1.$

(17)

Setting these partial derivatives to zero, we get the following set of M + 1 equations:

$\{\begin{array}{l}2{\gamma }_m{\displaystyle \sum _{k=1}^M{\gamma }_k{}^2{\text{tr}}_{m,k}}+\alpha =0,\hspace{1em}m=1,\mathrm{...},M\\ {\displaystyle \sum _{m=1}^M{\gamma }_m}-1=0\end{array}$

(18)

We use Newton's iteration method to solve these nonlinear equations. Let

$\mathbb{F}(\stackrel{\sim }{\gamma })=\left[\begin{array}{c}2{\gamma }_1{\displaystyle \sum _{k=1}^M{\gamma }_k^2{\text{tr}}_{1,k}}+\alpha \\ ⋮\\ 2{\gamma }_M{\displaystyle \sum _{k=1}^M{\gamma }_k^2{\text{tr}}_{M,k}}+\alpha \\ {\displaystyle \sum _{m=1}^M{\gamma }_m}-1\end{array}\right],$

(19)

where $\stackrel{\sim }{\gamma }={({\gamma }_1,\dots ,{\gamma }_M,\alpha )}^{\mathrm{T}}$

Then the iteration formula is

${\stackrel{\sim }{\gamma }}^{(i+1)}={\stackrel{\sim }{\gamma }}^{(i)}-{[{\mathbb{F}}^{\prime }({\stackrel{\sim }{\gamma }}^{(i)})]}^{-1}\mathbb{F}({\stackrel{\sim }{\gamma }}^{(i)}),$

(20)

where ${[{\mathbb{F}}^{\prime }({\stackrel{\sim }{\gamma }}^{(i)})]}^{-1}$ is the Jacobian matrix of $\mathbb{F}(\stackrel{\sim }{\gamma })$ at ${\stackrel{\sim }{\gamma }}^{(i)}={({\gamma }_1^{(i)},\mathrm{...},{\gamma }_M^{(i)},{\alpha }^{(i)})}^{\mathrm{T}}$ , $i=0,1,2,\mathrm{...}$ .

3.5 Weight optimization procedure

Based on the analysis above, the detailed weight optimization procedure for MKFD is described as follows.

Input : K^(m) = [k^(m) (x _r , x _t ,)] _N × N, m = 1,…,M, i.e., M base kernel matrices.

Output : γ = (γ₁,…, γ _M )^T, with γ² _m being the weight of base kernel K^(m)

S1. Given ε > 0. Initialize iteration counter i = 0 and $i=0$ and $({\gamma }_1^{(0)},\mathrm{...},{\gamma }_M^{(0)},{\alpha }^{(0)})$ , subject to ${\displaystyle {\sum }_{m=1}^M{\gamma }_m^{(0)}}=1$

S2. Using the diagonalization strategy of MKFD with the constructed multiple kernel matrix $K={\displaystyle \sum _{m=1}^M{\gamma }_m^{(i)}{}^{ 2}{K}^{(m)}}$ , find an optimal projection matrix in the i -th iteration

$W{*}^{(i)}={\mathrm{\Phi }}_b{G}^{(i)}=\arg \underset{W}{\max }{J}^{\mathrm{\Phi }}(W);$

S3. Regarding G⁽ⁱ⁾ as constant, construct the constrained optimization

S4.

$\begin{array}{c}\underset{\gamma }{\max }F(\gamma )=\text{tr}(G^{(i)}{}^{\mathrm{T}}P{P}^{\mathrm{T}}{G}^{(i)}-G^{(i)}{}^{\mathrm{T}}Q{Q}^{\mathrm{T}}{G}^{(i)}),\\ \mathrm{s}.\mathrm{t}.1^{\mathrm{T}}\gamma ={\displaystyle \sum _{m=1}^M{\gamma }_m}=1,\end{array}$

and calculate matrix (tr _m,k )M × M' where

$\begin{array}{c}{\text{tr}}_{m,k}=\text{tr}((G^{(i)}{}^{\mathrm{T}}{P}^{(m)}{P}^{(k)}{}^{\mathrm{T}}{G}^{(i)}+G^{(i)}{}^{\mathrm{T}}{P}^{(k)}{P}^{(m)}{}^{\mathrm{T}}{G}^{(i)})\\ -(G^{(i)}{}^{\mathrm{T}}{Q}^{(m)}{Q}^{(k)}{}^{\mathrm{T}}{G}^{(i)}+G^{(i)}{}^{\mathrm{T}}{Q}^{(k)}{Q}^{(m)}{}^{\mathrm{T}}{G}^{(i)})),\\ m,k=1,\mathrm{...},M.\end{array}$

S5 (Update weights) Compute ${\stackrel{\sim }{\gamma }}^{(i+1)}$ from ${\stackrel{\sim }{\gamma }}^{(i)}$ using Eqs. (19)(20). If $\Vert {\stackrel{\sim }{\gamma }}^{(i+1)}-{\stackrel{\sim }{\gamma }}^{(i)}\Vert >\epsilon$ , then $i=i+1$ , go to S2; else stop.

In our experiments reported in Section 4, for the initial value, considering Eq. (18), we set ${\gamma }_1^{(0)}=\mathrm{...}={\gamma }_M^{(0)}=\frac{1}{M}$ , $\alpha =-\frac{2}{M^4}{\displaystyle \sum _{m=1}^M{\displaystyle \sum _{k=1}^M{\text{tr}}_{m,k}}}$ , and ε is set to $5e-3$ .

4. Experiments

To evaluate the performance of our MKFD for face recognition, we have made experimental comparisons with KDDA based on single kernels, in terms of low-dimensional representation and image recognition. Images are from two face databases, namely the FERET and the CMU PIE databases.

In our experiments, three base kernels ( $M=3$ ) are adopted to construct the multiple kernel function: linear kernel $k_1(x_i,x_j)=x_i{}^{\mathrm{T}}{x}_j$ , Gaussian RBF kernel $k_2(x_i,x_j)=\exp (-\frac{{\Vert x_i-x_j\Vert }^2}{2{\sigma }^2})$ where σ is set to the average value of all the original sample distances $\frac{1}{N(N-1)}{\displaystyle {\sum }_{i=1}^N{\displaystyle {\sum }_{j=1}^N\Vert x_i-x_j\Vert }}$ , and polynomial kernel $k_1(x_i,x_j)={(x_i{}^{\mathrm{T}}{x}_j+1)}^d$ where d is set to 0.5. Thus the multiple-kernel is $k(x_i,x_j)={\displaystyle \sum _{m=1}^3{\gamma }_m{}^2{k}_m(x_i,x_j)}$ , with ${\displaystyle \sum _{m=1}^3{\gamma }_m}=1$ . We demonstrate the effectiveness of the multiple-kernel by comparing its performance with the single base kernels.

4.1 Face image datasets

From the FERET database [16], we select 72 people, with 6 frontal-view images for each individual. Face image variations in these 432 images include illumination, facial expression, wearing glasses, and aging. All the images are aligned by the centers of the eyes and the mouth and then normalized with a resolution of 92 × 112. The pixel value of each image is normalized between 0 and 1. The original images with resolution 92 × 112 are reduced to wavelet feature faces with resolution 49 × 59 after 1-level Daubichies-4 (Db4) wavelet decomposition. Images from one individual are shown in Fig. 1.

Figure 1.

Images of one person from the FERET database

In the CMU PIE face database [17], there are totally 68 people, and each person has 13 pose variations ranged from the full right profile image to the full left profile image and 43 different lighting conditions, 21 flashes with ambient light on or off. In our experiments, for each person, we select 56 images including 13 poses with neutral expression and 43 different lighting conditions in the frontal view. For all frontal-view images, we apply alignment based on two eye center and nose center points, and no alignment is applied on the other images with poses. All the segmented images are rescaled to the resolution of 92 × 112, and then reduced to wavelet feature faces with resolution 49 × 59 after 1-level Daubichies-4 (Db4) wavelet decomposition. Some images of one person are shown in Fig. 2.

Figure 2.

Some images of one person from the CMU PIE face database

4.2 Distribution of extracted features

This section aims to provide insights on how the proposed MKFD simplifies the face pattern distribution, compared with KDDA based on single kernels, when the patterns are subject to pose and illumination variations.

We select five subjects, 56 images per subject, with varying pose and illumination, from our CMU PIE face dataset determined above (56 × 5 = 280 images in all). Four types of feature bases are generalized from the images by utilizing KDDA with linear kernel, KDDA with Gaussian RBF kernel, KDDA with polynomial kernel, and our MKFD, respectively. In the sequence, all the 280 images are projected onto the four subspaces. For each image, its projections in the first two most significant feature bases of each subspace are visualized in Fig. 3.

Figure 3.

Distribution of 280 images of five subjects under varying pose and illumination in four types of subspaces

Fig. 3(a)–(d) depict the first two most discriminant features extracted by utilizing KDDA with linear kernel, KDDA with Gaussian RBF kernel, KDDA with polynomial kernel and MKFD, respectively. Obviously, our MKFD extracts the most discriminant features.

4.3 Recognition results

This section reports the recognition results of MKFD and KDDA with single kernels on the FERET and the CMU PIE datasets. For each subject in the FERET dataset, we randomly select n (n = 2 to 5) out of 6 images for training, with the rest for testing. In the CMU PIE dataset, the number of randomly selected training images is ranged from 10 to 18 out of 56 for each individual, while the rest are testing images. The average recognition accuracies over 10 runs on the FERET and CMU PIE datasets are shown in Fig. 4(a)–(b), respectively.

Figure 4.

Comparison of accuracies obtained by MKFD and KDDA with single kernels

Table 1 shows the average and standard deviation of the accuracies for FERET (n = 3: 3 images per subject for training with the rest for testing) and CMU-PIE (n = 14: 14 images per subject for training with the rest for testing), respectively.

Table 1.

Performance comparison between MKFD and KDDA with single kernels

Type of kernel		FERET (n=3)	CMU PIE (n=14)
Linear kernel	Mean Std	84.94% 0.015	71.79% 0.014
RBF kernel	Mean Std	73.84% 0.040	69.03% 0.010
Polynomial kernel	Mean Std	84.80% 0.017	76.50% 0.010
Multi-kernel with same weights	Mean Std	80.84% 0.030	71.81% 0.014
Multi-kernel with optimized weights	Mean Std	86.34% 0.015	78.74% 0.007

From the results in Fig. 4 and Tables 1, it can be seen that, the blending of multiple kernels in the proposed MKFD can achieve higher accuracies than any of the three single kernels, but a simple summation of multiple kernels is hardly a good idea for improving the classification performance, while our constructed multiple kernel function with optimized kernel weights leads to enhanced performance.

Note that in the experiments, neither the parameter for RBF kernel nor the one for polynomial kernel is optimally selected, and even linear kernel has no parameter. This means that, to a certain extent, the multiple kernel function relaxes parameter selection about the base kernels.

5. Conclusion

In this paper, on the assumption that multiple kernels can characterize geometrical structures of the original data from multiple views which can complement to improve recognition performance, we apply kernel-based FDA with multiple kernels, which we call MKFD, to recognition of face images under variations of pose, illumination, facial expression, etc. The constructed kernel for MKFD is a linear combination of several base kernels with a constraint on their weights. By maximizing the margin maximization criterion, we propose an iterative scheme based on the method of Lagrange multipliers for the weight optimization, which yields updated kernel weights resulting in high recognition accuracy on the FERET and CMU PIE face database, compared with the single kernels. The experiments also demonstrate that the multiple kernel function relaxes parameter selection to some extent.

It is important to point out that the proposed weight optimization scheme is generic and, with minor modifications, can be applied to all kernel-based FDA algorithms.