Sage Journals: Discover world-class research

Abstract

Multi-objective optimization and gray association are classical technologies in intelligent system. In this paper, a novel multi-focus image fusion method based on the technology is proposed. First, the input images are decomposed into low-frequency subband and high-frequency subbands using shift invariant Shearlet transform. Second, low-frequency subbands are fused by the weighted average fusion rule. The weight can be adaptively obtained by vector-evaluated quantum-behaved particle swarm optimization and gray relation analysis. The similarity of a corresponding region-based fusion rule is proposed and used to integrate high-frequency subbands. Last but not least, fusion image is obtained by inverse SIST. Visual and statistical analyses demonstrate that the fusion quality can be significantly improved by the proposed method.

Keywords

Vector-evaluated quantum-behaved particle swarm optimization gray relational analysis region entropy component analysis multi-focus image fusion

Introduction

Multi-sensor image fusion has become a hot topic in the area of information fusion in the past few years. The contents of multiple images tend to be complementary. Image fusion is an effective technique to combine complementary information from multiple images into a fused image, which is very useful for human visual or machine perception.¹ For example, optical imaging cameras cannot capture objects at various distances all in focus. To resolve this problem, image fusion can fuse several images with different sharp parts to get one image with all the objects focused. The basic idea of performing multi-focus image fusion is to choose the clearer image pixels (or blocks/regions) from source images or adaptively average observed images according to their respective sharpness, to construct the fused image.

Basically, there are two main methods for multi-focus image fusion. One is the spatial domain-based method, which select pixels or regions from clear parts in the spatial domain to compose fused images.² The simplest fusion method in spatial domain is to take the average of the source images pixel by pixel.³ The main drawback is that all information within the images are treated in the same way. To overcome this drawback, a normalized weighted aggregation approach is proposed for image fusion.⁴ However, pixel-based fusion methods come with several undesired side effects including reduced contrast and sensitive to noise or mis-registration. To improve the quality of fused image, some researchers proposed to fuse images based on blocks or segmented region instead of single pixels.^5–7 The key point of this kind of methods is how to design reasonable focus measures to choose the clearer part from source images to construct the fused image.⁸ Li et al.⁵ used spatial frequency (SF) to measure the clarity of image blocks. Miao and Wang⁹ and Eltoukhy and Kavusi¹⁰ used energy of image gradients to measure the activity. In addition, variance, energy of image gradient (EOG), Tenenbaum’s algorithm and energy of Laplacian (EOL)⁸ of the image are often used. However, the above-mentioned methods are easy to introduce blocking effect, and the boundary of focused regions cannot be accurately discriminated.

The other is transform domain-based fusion method, which fuse images with certain frequency or time-frequency transform.² As for this kind of methods, source images are projected onto localized bases and transformed coefficients are obtained.³ The salient features provided by transformed coefficients are used to construct the fused image. The multi-scale transform (MST) is often used in image fusion, which can efficiently avoid blocking effect because the processing of fusion is performed in frequency domain. Commonly used MST tool include the Laplacian pyramid,¹¹ wavelet,^12,13 Curvelet,¹⁴ Contourlet¹⁵ and nonsubsampled Contourlet transform (NSCT)² and Shearlet et al.^16,17 As we all know, one of the well-known MST methods for image fusion is wavelet. The wavelet-based statistical feature can be used to sharpness measure.¹⁸ Wavelet is good at isolating the discontinuities at object edges, but cannot effectively represent the ‘line’ and the ‘curve’ discontinuities. In addition, Wavelet decomposes image in only three direction subbands and thus cannot represent the directions of edges accurately. So wavelet-based fusion methods cannot preserve the salient features of source images very well and probably introduce some artifacts into the fused image. Concerning the curvelet transform, however, it does not build directly on the discrete domain and provide a multi-resolution representation of the geometry. As for the contourlet transform, the shift-invariance is lost while the NSCT is of high-time cost. SIST is one of the state-of-the-art MST means,¹⁷ which has a rich mathematical structure similar to wavelet and the true 2-dimensional (2D) sparse representation for images with edges. It is more efficient for mathematics analysis and has been applied in image fusion.

Except for choosing MST tool, the other important thing is to design fusion rule. Considering the characteristic of human visual perception and the relationship among pixels, novel fusion rules combining intelligent algorithm and the similarity of regions are developed in this paper. Firstly, the source images are decomposed by using SIST. Secondly, fusion rules are designed for low-frequency subband and high-frequency subbands, respectively. The fusion rule of low-frequency subbands based on multi-objective optimization and gray association is designed. The fusion rule of high-frequency subbands based on the similarity of corresponding region is employed. If the similarity of corresponding region is greater than the threshold value, a fusion rule based on window is employed. Otherwise, a fusion rule based on region is performed. Finally, fused image is obtained by taking inverse SIST transform. The proposed method adequately considers the salient feature of image, and exploits the characteristic of region to design fusion strategy. According to fusion purpose, the fusion evaluation metrics are selected as multi-object functions of vector-evaluated quantum-behaved particle swarm optimization (VEQPSO) to realize the fusion of low frequency subbands. Experimental results show that the approach is capable of fusing images with higher image quality.

The remaining sections of this paper are organized as follows: Partition the source images presents how to get image regions. The next section describes the proposed image fusion method. Experimental results and discussion are given in the following section. Conclusions are given in the final section.

Partition the source images

The source images are partitioned into region because regions rather than arbitrary pixels are more meaningful in image fusion. To obtain the region map of source images, the average image of them is calculated.^7,19 It is divided into $m \times m$ small windows. Let $num$ be the number of windows. A feature vector $f_{w} \to ({\vec{f}}_{w} \in R (7 + m), 1 \leq w \leq num)$ is then extracted from each window. Four of $f_{w} \to$ are the texture features, the fifth feature is the mean of pixels in a window. The other two are the x-coordinate and y-coordinate on the top left corner of a window, which denotes the spatial location. The other m is the 2D entropy principal component of a window, which eliminates data redundancy and reflects gray distribution and represents entropy information of a window. Finally, the feature vectors ⃗f are obtained and normalized to ^f.

Texture features

Four of $f_{w} \to$ are the texture features,²⁰ which are extracted by SIST transform applied to the average image. After a one-level SIST transform, the average image is decomposed into four frequency subbands. Without loss of generality, we denote the coefficients in one subband as ${c_{k, l}, c_{k, l + 1}, c_{k + 1, l}, \dots, c_{k + M - 1, l + N - 1}}$ . One feature is obtained by

f = (\frac{1}{m \times m} \sum_{wi = 0}^{m - 1} \sum_{wj = 0}^{m - 1} c_{k + wi, l + wj}^{2}) \frac{1}{2}

(1)

The other three features are computed similarly.

2D entropy principal component

In this paper, 2D entropy principal component analysis (2DECA) method is applied to the feature extraction of a window. Instead of selecting features based on the cumulative variance contribution, the features are extracted from the 2DECA subspace based on the Renyi entropy contribution.^21,22 The 2DECA makes full use of the 2D information of window and avoids reshaping a window matrix into a window vector. Those principal axes contributing most to the Renyi entropy estimate clearly carry most of the information of input space data set. The 2DECA algorithm is given as follows.

Input the dataset $F = [F_{1}, F_{2}, \dots, F_{w}, \dots F_{num}]$ , whereF_wis a $m \times m$ matric.

Calculate the average image of all training samples: $M = \frac{1}{num} \sum_{w = 1}^{num} F_{w}$ .

Calculate the covariance matrix: $C = \frac{1}{num} \sum_{w = 1}^{num} (F_{w} - M) T (F_{w} - M)$ .

Calculate the eigenvalues and eigenvectors of C, which are denoted as $λ = [λ_{1} \dots \dots λ_{num}]$ and $E = [e_{1} \dots \dots e_{num}]$ , respectively.

Calculate the estimate value of entropy: $sp = \frac{1}{num 2} \sum_{w = 1}^{num} (\sqrt{λ_{w}} e_{w}^{T} l) 2$ . Features extraction: Select eigenvectors corresponding to the entropy contribution as the projection.

The window sample $W$ and the optimal projection vectors of 2DECA $(x_{1}, x_{2}, \dots x_{d})$ are used for feature extraction.

Projection: $V_{K} = W X_{K}, K = 1 \dots . . d$ .

Output: $V_{K}$ .

v₁is the first entropy principal component, the size of v₁is $1 \times m$ .Then, v₁is looked as the feature of window, which reflects the gray distribution and entropy of window.

Image segmentation by FCM

Image segmentation is an essential step in the proposed method. Fuzzy clustering techniques have been effectively used in image processing, pattern recognition and fuzzy modeling. FCM adopts the iterative algorithm to optimize objective function.²³

$μ_{rw}$ ( $1 \leq r \leq c$ , $1 \leq w \leq num$ ) and c can be obtained by using FCM algorithm, where $μ_{rw} \in [0, 1]$ denotes the membership of the feature vector ${\overset{\land}{f}}_{w}$ in therth cluster and c is the number of clusters. Considering the limitation of space, the details of FCM are omitted in this paper which can be found in Nikhil et al.²³

The clusters can be obtained by FCM as follows:

If $μ_{rw} > μ_{r' zw}$ ( $r = 1, 2, \dots, c$ , $r' = 1, 2, \dots, c$ , $r \neq r'$ ), then the vector ${\overset{\land}{f}}_{w}$ belongs to the rth cluster. Consequently, the feature vectors ^f is partitioned into c regions.

The proposed fusion algorithm

To improve the quality of fused image, multi-objective optimization and gray association are used to fuse the low-frequency subband coefficients, and region similarity is employed to guide the fused process of high-frequency subband coefficients.

The fusion rule of low-frequency subband coefficients

The coefficients in the coarsest scale subband represent the approximation component of the source image. The average weighted fusion rule is employed to fuse the low-frequency subband coefficients

{LF}_{k} (i, j) = λ_{1} {LA}_{k} (i, j) + λ_{2} {LB}_{k} (i, j), λ_{1} + λ_{2} = 1

(2)

where

λ_{1}

and

λ_{2}

are adaptive obtained by VEQPSO and gray relation analysis. The construction of multi-object functions is based on fusion purpose, which will improve the contrast of fused image and hold more useful information of the source images.

In VEQPSO, $λ_{1}$ and $λ_{2}$ are denoted as decision variables. The optimal weight is given by evolutionary iterations. The multi-objective optimization is determined by the evaluation indexes of multi-focus image fusion. The VEQPSO algorithm is proposed in Omkar et al.,²⁴ which borrows concepts from Schaker’s vectors evaluated genetic algorithm (VEGA). The VEQPSO algorithm initializes a particle in the decision variable space and makes it eventually fall into the Pareto optimal solutions via multiple objective functions collaborative guiding the flight of particle in decision variable space. To balance the multi-objective control to particles flying, gray relation analysis is used to calculate the control weight. Gray relation analysis is not subject to the number and distribution pattern of samples.²⁵ Moreover, the overall comparison mechanism makes it stronger discrimination ability to research all kinds of complex relationships in system.

In view of the fact that QPSO is the basis of VEQPSO, it will be introduced firstly.

QPSO is an excellent evolutionary search technology. The moving process of particles is described as follows

mbest = \frac{1}{M} \sum_{i = 1}^{M} P_{i} = (\frac{1}{M} \sum_{i = 1}^{M} P_{i 1} \dots \dots \frac{1}{M} P_{ij})

(3)

{PG}_{ij} = f * P_{ij} + (1 - f) * {Pg}_{j} f = rand

(4)

x_{ij} = {PG}_{ij} \pm α * | {mbest}_{j} - x_{ij} | * \ln (\frac{1}{u}) u = rand

(5)

where

x_{ij}

is the position vector of the ith particle and vector P_i is the best previous position (the position giving the best fitness value) of particle i, and vector

{Pg}_{j}

is the position of the best particle among all the particles in the population.

mbest

is defined as the average best position of all particles P_i.

{PG}_{ij}

is the random points between

P_{ij}

and

{Pg}_{j}

.αis the contraction expansion coefficient and MAXITER is the maximum number of iterations.

Next, the VEQPSO based on gray relation analysis is given as follows.

Initialize swarms. Suppose the number of object is N, then randomly generate N swarms $S = [s_{1}, \dots s_{r'}, \dots s_{N}]$ . For $1 : N$

Calculate the global best particle ${Pg}_{r' j}$ of each swarm by using QPSO.

After getting ${Pg}_{r' j}$ , the social shared information of particle swarm is calculated based on gray relation analysis. ${Pg}_{j}$ is employed as fused weight and then the fused result of low-frequency subband can be obtained. Comparison sequence $x_{r'} (k)$ is composed by the objective function values of fused result. Reference sequence $x_{0} (k)$ composed by the optimal value in each column of $x_{r'} (k)$ . Calculate the degree of gray relation ( $R_{0 r'}$ ) between $x_{r'} (k)$ and $x_{0} (k)$ . The social shared information of particle swarm can be obtained by:

Pg = \sum_{r' = 1}^{N} {wp}_{r'} {Pg}_{r' j}, {wp}_{r} = \frac{R_{0 r'}}{\sum_{r' = 1}^{N} R_{0 r'}}

(6)

Calculate the mbest of each particle swarm. Then, PG_ij can be gotten.

Use the global best particle ( $Pg$ ) and $mbest$ to calculate ${PG}_{ij}$ and the particle positions ( $x_{ij}$ ).

The whole process 2–5 is repeated until maximum number of iterations. Finally, a group of Pareto optimal solutions selected from each of swarm is outputted as the optimization results.

The fusion rule of high-frequency subband coefficients

Suppose that the feature matrix of rth region can be denoted as $f_{z}^{r} = ({\overset{\land}{f}}_{1, z}^{r}, {\overset{\land}{f}}_{2, z}^{r} \dots, {\overset{\land}{f}}_{w', z}^{r}, \dots {\overset{\land}{f}}_{num_r, z}^{r})$ ( $r = 1, 2, \dots, N_cluster$ , $1 \leq w' \leq num_r$ , $z = AorB$ . The size of $f_{z}^{r}$ is $(7 + m) \times num_r$ , where $num_r$ is the window number of rth region.

The similarity of corresponding window in the corresponding region is given as follows²⁶

{WS}_{w'}^{r} = 1 - \frac{| {\overset{\land}{f}}_{w', A}^{r} - {\overset{\land}{f}}_{w', B}^{r} |}{| {\overset{\land}{f}}_{w', A}^{r} | + | {\overset{\land}{f}}_{w', B}^{r} |}

(7)

Let RS r = [{WS}_{1}^{r}, \dots {WS}_{w'}^{r}, \dots {WS}_{num_r}^{r}], E_{w', z}^{r, k} = \sum_{p = 1}^{3} \sum_{n \in W_{w'},} | y_{r}^{k} (n | p) | 2

(8)

If E_{w', A}^{r, k} \geq E_{w', B}^{r, k}, {WE}_{w'}^{r, k} = E_{w', A}^{r, k}, else {WE}_{w'}^{r, k} = E_{w', B}^{r, k}

(9)

Then RE r = [{WE}_{1}^{r, k}, \dots {WE}_{w'}^{r, k}, \dots {WE}_{num_r}^{r, k}]

(10)

Let sum_E = sum (RE r), then S_{AB}^{r} = \sum_{w' = 1}^{num_r} \frac{{WE}_{w'}^{r, k}}{sum_E} \times {WS}_{w'}^{r}

(11)

where

y_{r}^{k} (n | p)

is the pth direction SIST coefficients of rth region at kth level. nis the serial number of coefficient in a window.

E_{w', z}^{r, k}

is the energy of

w'

th window in rth region of source image z. k denotes the decomposition level, which is defined as 1 in equation (10).

E_{w', z}^{r, k}

reflects the salience of window. Let

RE r

be the reference vector of window energy, and the weight of each window in the region can be calculated.

S_{AB}^{r}

denotes the similarity of rth corresponding region.

The fusion rule of high-frequency subband coefficients can be given by

Else

y_{w', F}^{r, k} (n | p) = w_{A} y_{w', A}^{r, k} (n | p) + w_{B} y_{w', B}^{r, k} (n | p), w_{A} = 1 - w_{B} = d_{w', A}^{r, k, p} (n | p) = \frac{E_{w', A}^{r, k, p}}{E_{w', A}^{r, k, p} + E_{w', B}^{r, k, p}}

(13)

where Fis the fused image,

ω_{A}

and

ω_{B}

are the weighting factor (

ω_{A} = 1 - ω_{B} = d_{r, A}^{k} (n | p)

), Tis the threshold. If

S_{AB}^{r}

less thanT, then the corresponding region is complementary. Otherwise, it is redundant. According to the attribution of corresponding region, different fusion rules are designed. If

S_{AB}^{r} \leq T

, the process of fusion is performed on region, otherwise, on each window in the region.

The procedure of the proposed fusion method

The scheme of the proposed method is depicted in Figure 1, and the fusion process is given in details as below.

Decompose the source images AandB by using SIST.

The average weighted fusion rule is employed to fuse the low-frequency subbands. The weight can be adaptive obtained by VEQPSO and gray relation analysis.

The average image of Aand B is partitioned into windows and then the feature vectors are extracted from each window. FCM algorithm is used to cluster the feature vectors. The segmentation results are mapped intoAandB.

The fusion rule of high-frequency subbands is designed based on the similarity of corresponding region.

The fused image F is obtained by inverse SIST.

Figure 1.

The scheme of the proposed method.

Experimental results and analysis

In this section, the ‘goldhill’ and ‘book’ images are employed to demonstrate the effectiveness of our proposed method. The source images are obtained from the website http://www.imagefusion.org/. The size of images are $256 \times 256$ and $256 \times 192$ , respectively. The pyramid filter of SIST is set as ‘maxflat (maximally flat filter)’ and the source images are decomposed into four levels with the number of directions 4, 4, 8, 8. The threshold of proposed method is experimentally set as $T = 0.8$ in Figures 2 and 3. The particle number of each of particle swarm is 30 and iteration number is 30. The setting of above parameters is experimentally selected. Without loss of generality, the number of multi-objective function is defined as 2 in our experiment. The objective function can be selected according to the practical application. Peak signal-to-noise ratio (PSNR) and average gradient (grad) are used as the multi-objective function in our experiment.¹⁹

Figure 2.

The fusion results of goldhill images. (a) Source image A; (b) source image B;(c)the average weighted fusion method;(d) PCA based method;(e) DWT based method; (f) morphological wavelets based method; (g) the normalized weighted aggregation based method;(h) the proposed fusion method with 2DPCA; (i) the proposed method;

Figure 3.

The fusion results of book images. (a) source image A; (b) source image B;(c) the average weighted fusion method;(d) PCA based method;(e) DWT based method; (f) morphological wavelets based method; (g) the normalized weighted aggregation based method;(h) the proposed fusion mthod with 2DPCA; (i) the proposed method;

The proposed fusion algorithm compared with other typical fusion methods include: average weighted-based method, PCA-based method,²⁷ DWT-based method,²⁸ morphological wavelets-based method (Ishita De’s method²⁹), the normalized weighted aggregation-based method (Tian’s method¹⁸) and the proposed fusion method using 2DPCA. Some objective evaluation indices are employed to evaluate the quality of fused image, which includes: mean (M), standard deviation (SD), entropy (E), average gradient (G), spatial frequency (SF) and mutual information (MI).¹⁹ The high values of the above indexes means the better fusion quality.

In this section, two sets of images with perfect registration are used to evaluate the proposed fusion algorithm. As shown in Figure 2(a and b), each image contains multiple objects at different distances from the camera. The focus in Figure 2(a) is on the forest, while that in Figure 2(b) is on the house. Figure 2(c–g) are the fused images by using the above typical fusion methods. Figure 2(i) is the fusion result obtained by using the proposed method. In this experiment, the size of window is

3 \times 3

and grad is selected as major objective function in the selection of optimal solution set. Figure 2(h) is the fusion result obtained by the proposed method using 2DPCA, the parameters of which are the same as the proposed method. The effects of different window size and different major objective function on the quality of fused image are investigated. It is obvious that Figure 2(i, h and g) are clearer than Figure 2(f), and Figure 2(f) are clearer than Figure 2(c, d and e). The sharpness of fused image denotes the capability of reserving useful information in the fusion process. For further comparison, six objective criteria are used to compare the fusion results. The values of six indexes are listed in Table 1. The performance of Figure 2(d and e) are closed, which are better than Figure 2(c). Figure 2(g) has obvious improvement compared to Figure 2(f) in terms of G, SF and MI. As can be seen from Table 1, we find that the setting of parameters influence the quality of fused image. When grad is selected as major objective function in the selection of optimal solution, the quality of fused image is better than PSNR. When the size of window is defined as 3

\times 3

, the quality of fused image is the best. Figure 2(h) is the fused result obtained by using 2DPCA to replace 2DECA of the proposed method. The project axes of 2DECA with maximum contribution to the Renyi entropy estimation is defined as the principal axes of 2DECA. Using it to dimensionality reduction, the information of data set can be preserved as much as possible. Therefore, more reasonable regional segmentation result can be obtained by introducing 2DECA to extract the feature of data. As a result, Figure 2(i) is superior to Figure 2(h) in terms of all evaluation indexes.

Table 1.

Values of the object evaluations for fusion images in Figure 2 (the best values are in bold).

Images/statistics	M	SD	E	G	SF	MI
Figure 2(c)	112.1919	46.3811	5.1408	8.9277	11.8474	5.0108
Figure 2(d)	112.4014	47.3861	5.1417	9.4505	12.9196	7.2452
Figure 2(e)	112.1903	47.3854	5.1415	9.4994	12.9943	7.2661
Figure 2(f)	112.1237	47.4591	5.1575	10.8646	14.5768	7.3643
Figure 2(g)	112.2178	48.1011	5.1600	11.9917	16.7668	7.6898
Figure 2(h): (3 × 3) grad (2DPCA)	112.1981	48.2168	5.1663	12.2756	16.8021	7.6942
Figure 2(i): ( $3 \times 3$ ) PSNR	112.2164	48.2156	5.1651	12.1587	16.7835	7.6754
Figure 2(i): ( $3 \times 3$ ) grad	112.1984	48.2368	5.1683	12.2894	16.8376	7.7195
Figure 2(i): ( $5 \times 5$ ) PSNR	112.1972	48.2015	5.1645	12.0472	16.5663	7.6143
Figure 2(i): ( $5 \times 5$ ) grad	112.1974	48.1963	5.1657	12.1753	16.6768	7.6367
Figure 2(i): ( $7 \times 7$ ) PSNR	112.1936	48.1637	5.1635	11.9993	16.4835	7.5439
Figure 2(i): ( $7 \times 7$ ) grad	112.1887	48.1565	5.1652	11.9936	16.5973	7.5235

To further evaluate the fusion performance, the second experiment is performed on other set of multi-focus images. As shown in Figure 3(a) and (b), each image contains multiple objects at different distances from the camera. The focus in Figure 3(a) is on the book titled openGL, while that in Figure 3(b) is on the other book. The fusion results obtained by the above typical fusion methods and our proposed method are shown in Figure 3(c–i), respectively. For clearer comparison, the source images and fused images are given in Figure 3(a–i). As can be seen from Table 2, with different fusion methods appears different performance. Although MI of the normalized weighted aggregation-based method is slightly larger than that of the proposed method, the visual appearance of Figure 3(g) is obviously blur. By the comparison, it can be concluded that Figure 3(e) is the worst fused result and Figure 3(i) is the best one when the size of window is defined as

5 \times 5

and PSNR is selected as major objective function in the selection of optimal solution set. The qualities of fused images are expressed in descending numerical order, that is, Figure 3(i), Figure 3(h), Figure 3(g), Figure 3(f), Figure 3(d), Figure 3(c), Figure 3(e). In terms of the visual effect and the objective evaluation results, we can draw the conclusion that the proposed method is with the highest performance and preserves the useful information better than the other methods.

Table 2.

Values of the object evaluations for fusion images in Figure 3 (the best values are in bold).

Images/statistics	M	SD	E	G	SF	MI
Figure 3(c)	91.3820	54.7740	5.0391	12.0736	21.3832	6.9688
Figure 3(d)	91.2056	54.7994	5.0395	12.0971	21.3869	7.0322
Figure 3(e)	91.1275	52.6717	5.1482	11.2686	16.4578	4.3043
Figure 3(f)	91.2165	54.9428	5.1238	12.8733	23.7653	5.8364
Figure 3(g)	91.3822	55.0616	5.0308	13.3385	24.1416	6.9975
Figure 3(h): (5 × 5) PSNR (2DPCA)	91.4532	55.5674	5.1418	14.4543	25.2175	6.9732
Figure 3(i): ( $3 \times 3$ ) PSNR	91.4423	55.3841	5.1457	14.4552	25.1865	6.9557
Figure 3(i): ( $3 \times 3$ ) grad	91.2186	55.1842	5.1343	14.3576	25.0265	6.8321
Figure 3(i): ( $5 \times 5$ ) PSNR	91.4854	55.5897	5.1472	14.4625	25.2199	6.9764
Figure 3(i): ( $5 \times 5$ ) grad	91.3357	55.1953	5.1432	14.3998	25.2096	6.9315
Figure 3(i): ( $7 \times 7$ ) PSNR	91.4328	55.3867	5.1354	14.4561	25.1963	6.9513
Figure 3(i): ( $7 \times 7$ ) grad	91.3125	55.2324	5.1398	14.3853	25.1552	6.9176

Figure 2(a) source image A; (b) source image B; (c) the average weighted fusion method; (d) PCA-based method; (e) DWT-based method; (f) morphological wavelets-based method; (g) the normalized weighted aggregation-based method; (h) the proposed fusion method with 2DPCA; (i) the proposed method.

Figure 3(a) source image A; (b) source image B; (c) the average weighted fusion method; (d) PCA-based method; (e) DWT-based method; (f) morphological wavelets-based method; (g) the normalized weighted aggregation-based method; (h) the proposed fusion method with 2DPCA; (i) the proposed method.

Conclusions

In this paper, a new method based on multi-objective optimization and gray association is presented. The underlying advantages include: (1) The relationship between image fusion and multi-objective optimization method is constructed (2) The weights of objective functions are determined by gray association in the optimization process. (3) 2DECA has the advantage in information reservation, which is used to extract the feature of window and then is beneficial for getting the reasonable region. (4) Region-based fusion rule is designed, which can increase the reliability and robustness of the fusion method. The experimental results on several pairs of multi-focus images have demonstrated the superior performance of the proposed fusion method.

Footnotes

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the National Natural Science Foundation of P.R. China under grant no. 61103128, the China Postdoctoral Science Foundation under grant no. 2013M541601, Postdoctoral Research Funds of Jiangsu province under grant no. 1301079C, the Natural Science Foundation of Jiangsu Province of China under grant no. BK20151358, BK20151202 and the Fundamental Research Funds for the Central Universities JUSRP51618B. The Ministry of Housing and Urban-rural Development of the People's Republic of China under grant no. 2015-K8-035.

References

Shutao

Xudong

Kang

Jianwen

. Image matting for fusion of multi-focus images in dynamic scenes. Inform Fusion 2013; 14: 147–162.

Huafeng

Chai

Zhaofei

. Multi-focus image fusion based on nonsubsampled contourlet transform and focused regions detection. Optik 2013; 124: 40–51.

Burt PJ and Kolczynski TJ. Enhanced image capture through fusion. In: Proceeding(s) of the 4th international conference on computer vision, 1993, p.82.

Jing

Tian

Chen

Lihong

. Multi-focus image fusion using a bilateral gradient-based sharpness criterion. J Opt Commun 2011; 284: 80–87.

Kwok

Wang

. Combination of images with diverse focuses using the spatial frequency. Inform Fusion 2001; 2: 169–176.

Piella

. A general framework for multiresolution image fusion: from pixels to regions. Inform Fusion 2003; 4: 259–280.

Yang

. Multifocus image fusion using region segmentation and spatial frequency. Image Vision Comput 2007; 26: 971–979.

Wei

Huang

Zhongliang

Jing

. Evaluation of focus measures in multi-focus image fusion. J Pattern Recog Lett 2007; 28: 493–500.

Miao Q and Wang B. A novel adaptive multi-focus image fusion algorithm based on PCNN and sharpness. In: Proc of SPIE, 2005, vol. 5778, pp.704–712.

10.

Eltoukhy HA and Kavusi S. A computationally efficient algorithm for multi-focus image reconstruction. In: Proc of SPIE Electronic Imaging, 2003, pp.332-341.

11.

Jahard F, Fish DA, Rio AA, et al. Far/near infrared adapted pyramid-based fusion for automotive night vision. In: Proceeding(s) of IEEE 6th Int. Conf. on Image Processing and its Applications (IPA97), 1997, pp.886-890.

12.

. Contrast-based image fusion using the discrete wavelet transform. Opt Eng 2000; 39: 2075–2082.

13.

Mallat

. A theory for multiresolution signal decomposition: the wavelet representation. IEEE Trans Pattern Anal 1989; 11: 674–693.

14.

Candès

Demanet

Donoho

. Fast discrete curvelet transforms. SIAM Multisc Model Simul 2006; 5: 861–899.

15.

Vetterli

. The contourlet transform: an efficient directional multiresolution image representation. IEEE Trans Image Process 2005; 14: 2091–2106.

16.

Easley

Labate

Lim

. Sparse directional image representations using the discrete shearlet transform. Appl Comput Harmon 2008; 25: 25–46.

17.

Guo

Wang-QL Lambaste

. Wavelets with composite dilations and their MRA properties. Appl Computat Harmon Anal 2006; 20: 202–236.

18.

Jing

Tian

Chen

. Adaptive multi-focus image fusion using a wavelet-based statistical sharpness measure. Signal Process 2012; 92: 2137–2146.

19.

Xiaoqing

Luo

Xiaojun

. New metric of image fusion based on region similarity. Opt Eng 2010; 49: 047006–1-047006-13.

20.

Chen

Wang

. A region-based fuzzy feature matching approach to content-based image retrieval. IEEE Trans Pattern Anal Machine Intell 2002; 24: 1252–1267.

21.

Robert

Jenssen

. Kernel entropy component analysis. IEEE Trans Pattern Anal Machine Intell 2010; 32: 847–860.

22.

Xing Liu and Xiao-Jun Wu. ECA and 2DECA: entropy contribution based methods for face recognition inspired by KECA. In: International conference of soft computing and pattern recognition (SoCPaR), 2011, pp.544-549.

23.

Nikhil

Kuhu

James

. A possibilistic fuzzy c means clustering algorithm. IEEE Trans Fuzzy Sys 2005; 13: 517–530.

24.

Omkar

Khandelwal

Rahul

Ananth

TVS

. Quantum behaved particle swarm optimization (QPSO) for multi-objective design optimization of composite structures. Expert Sys Appl 2006; 36: 11312–11322.

25.

Deng

. Control problems of grey system. Sys Contr Lett 1982; 1: 82–94.

26.

Kim-Han

Thung

Raveendran

Paramesran

. Content-based image quality metric using similarity measure of moment vectors. Patt Recogn 2012; 45: 2193–2204.

27.

Kumar S, Senthil and Muttan S. PCA-based image fusion. In: Proceeding(s) of the SPIE, 2006, pp.62331T.

28.

Manjunath

Mitra

. Multisensor image fusion using the wavelet transform. Graph Models Image Process 1995; 57: 235–245.

29.

Ishita

Bhabatosh

Chanda

Buddhajyoti

Chattopadhyay

. Enhancing effective depth-of-field by image fusion using mathematical morphology. Image Vision Comput 2006; 24: 1278–1287.

Multi-objective optimization and gray association for multi-focus image fusion

Abstract

Keywords

Introduction

Partition the source images

Texture features

2D entropy principal component

Image segmentation by FCM

The proposed fusion algorithm

The fusion rule of low-frequency subband coefficients

The fusion rule of high-frequency subband coefficients

The procedure of the proposed fusion method

Experimental results and analysis

Conclusions

Footnotes

Declaration of conflicting interests

Funding

References