Fabric defect detection based on deep-handcrafted feature and weighted low-rank matrix representation

Abstract

In the process of textile production, automatic defect detection plays a key role in controlling product quality. Due to the complex texture features of fabric image, the traditional detection methods have poor adaptability, and low detection accuracy. The low rank representation model can divide the image into the low rank background and sparse object, and has proven suitable for fabric defect detection. However, how to further effectively characterize the fabric texture is still problematic in this kind of method. Moreover, most of them adopt nuclear norm optimization algorithm to solve the low rank model, which treat every singular value in the matrix equally. However, in the task of fabric defect detection, different singular values of feature matrix represent different information. In this paper, we proposed a novel fabric defect detection method based on the deep-handcrafted feature and weighted low-rank matrix representation. The feature characterization ability is effectively improved by fusing the global deep feature extracted by VGG network and the handcrafted low-level feature. Moreover, a weighted low-rank representation model is constructed to treat the matrix singular values differently by different weights, thus the most distinguishing feature of fabric texture can be preserved, which can efficiently outstand the defect and suppress the background. Qualitative and quantitative experiments on two public datasets show that our proposed method outperforms the state-of-the-art methods.

Introduction

Fabric defect detection plays an important role in the quality control of textile production. ¹ It is traditionally conducted through visual inspections by the skilled workers, which is time-consuming and highly subjective. Therefore, manual approaches cannot meet the industry requirements. The fabric defect detection based on machine vision has drawn more attention in recent years because it can improve the detection accuracy and efficiency. Currently, some detection systems have been applied in the textile process, such as EVS I-Tex2000, Barco Visions Cyclops, and MQT, and achieve the ideal detection result. However, the details of these fabric defect detection algorithms have not been disclosed because of the intellectual property constraints, moreover, the existing devices are lack of self-adaptability for many kinds of fabric image. Therefore, the fabric defect detection method is still a research hotspot.

The fabric defect detection methods based on machine vision should be adapt to the fabric images with different texture, from simple non-motif fabric patterns (twill and plain fabrics, as shown in Figure 1(a)) to the complex motif fabric patterns (dot-, box-, and star-patterned fabrics, as shown in Figure 1(b)–(d)). For the plain and twill fabrics, the background is homogeneous and the defect is salient. Therefore, it is relative easy to detect defect for these fabrics. Most of the exiting defect detection methods focus on the twill and plain fabrics, and achieve high detection rate. According to the adopted image processing technology, they can be classified into four categories: statistical analysis methods, ² frequency transform methods, ³ model based methods,^4,5 and dictionary learning methods. ⁶ These methods are lack of adaptivity, and cannot be extended to the patterned fabrics with complex texture.

Figure 1.

(a) Plain and twill, (b) star-patterned fabric, (c) box-patterned fabric, and (d) dot-patterned fabric.

The patterned fabrics are defined as fabrics with the repetitive patterned units in their designs. Even within the same type of patterned fabrics, the texture and the pattern sizes are different. Currently, some methods aiming at the patterned fabric defect detection have been proposed, such as the ELO rating (ER) method, ⁷ and wavelet-preprocessing golden image subtraction (WGIS), ⁸ and so on, which are performed in a supervised approach, require non-defective samples; In addition, the detection accuracy of these methods depends on the accurate partitions and the selected template, and is lack of the self-adaptivity.

Low-rank decomposition model aiming at recovering the underlying low-dimensional component of matrix can divide the data matrix into the redundant part which spanning several low-rank subspaces and the sparse part which is the outlier, has been successfully applied in object detection, image segmentation, image denoizing, etc.^9,10 For any kinds of fabric images, their defect-free regions are macro-homogeneous, and lay in a low-dimensional subspace, while the defective regions are salient and sparse. Therefore, the task of the fabric defect detection can better fit the low-rank decomposition model comparing with the traditional object detection, and several fabric detection methods based on low-rank decomposition have been proposed.

However, the method based on low rank decomposition implicitly assumes that the fabric image is composed of two parts (background part and defect part). In fact, the defect fabric can be regarded as the combination of background part, defect part, and noise part. Therefore, the details between the various parts are not well considered in the method based on low rank decomposition, and the recovery may not be accurate. To deal with mixed data matrix, noise and corrosion better, dictionary learning theory is introduced into low rank decomposition model to generate a more general model, which is denoted as low rank representation model. However, the performance of this kind of methods depends on the effective feature descriptor and low-rank representation model.

Because the convolutional neural network can automatically learn the representative features layer by layer through a continuous propagation channel, the features generated by the convolutional neural network are global, especially can better capture the overall context information, thus it is generally believed that the features obtained by convolution neural network (CNN) are superior to the traditional handcrafted descriptors. Meanwhile, the handcrafted feature can also better characterize the low-level texture feature, and is complementary to the high-level deep feature. ¹¹ Therefore, we propose the deep-handcrafted feature by combining the deep feature with handcrafted feature to further improve the characterization capability for the fabric image.

For the low rank representation models, they usually use nuclear norm to convex surrogate, and are solved by SVT algorithm. ¹² However, the SVT method treats each singular value equally, in fact, the larger singular value contains more information, and thus the solution should be improved by differently treat the singular value. The larger singular values corresponding to the main projection direction should be retained to the maximum extent to preserve the main information of background. For smaller singular values, more penalty should be given to reduce the influence of outliers. Therefore, the weighted low rank representation (WLRR) is adopted to separate the sparse defect from background.

In summary, a novel fabric defect detection method based on deep-handcrafted feature and weighted low rank representation (WLRR) is proposed. The main contributions of our method can be summarized as follows:

• (1) Because of the complexity of fabric texture, only the deep features or handcrafted features cannot fully characterize the fabric images. In this paper, a novel feature descriptor, denoted as deep-handcrafted features which are constructed by fusing the deep feature with handcrafted feature, is proposed to effectively represent fabric images.

• (2) The solution method of low-rank representation model equally treats each singular value, and may causes the sparse defect and background cannot be separated effectively. To address this issue, a weighted low rank representation model is proposed, and treats the different singular values of feature matrix differently, thus it can preserve the main information of background and reduce the influence of outliers.

• (3) Extensive experiments are conducted on two public fabric datasets, and the results show that the proposed method has good detection performance for plain or twill fabrics with simple textures as well as for patterned fabrics with complex textures. Moreover, the proposed method is superior to the state-of-the-art methods.

The rest of the paper is arranged as follows. The section II briefly reviews the relevant methods, and the section III introduces the details of the method. In section IV, a comprehensive evaluation of the proposed method is conducted through experiments. Section V summarizes the research.

Related works

Fabric defect detection plays an important role in textile industry. Machine vision provides an ideal solution for this problem. Many researchers focus on this study, and proposed some effective methods. These detection methods based on machine vision can be divided into two categories: one is for fabrics with plain or twill texture, the other is for the fabrics with patterns. The methods for fabric with plain or twill texture include statistical-based method, spectral analysis-based method, model-based method, dictionary learning-based method.

The statistical method divides the test image into image blocks, and then calculates their texture statistics properties. ² The image blocks which occupy most of the same statistical properties are labeled as defect-free regions, and the others are marked as defect regions. The statistical methods include texton feature methods ¹³ and histogram method, ¹⁴ etc. However, these kinds of method are only applicable for the specific fabric texture, thus are lacking of adaptability.

The model-based methods model the fabric texture as a random process, and the defect detection is treated as a hypothesis testing problem based on the model. The commonly adopted models are Gaussian mixture model ¹⁵ and Gaussian Markov random field model, ¹⁶ etc. However, these methods often have high computational complexity, and poor performance for the defects with small size.

Dictionary learning methods first construct the dictionary from the test image, and then the defect free image is reconstructed using the dictionary. Finally, the defect detection is realized by subtracting the reconstructed defect free image and the test image. ⁶ However, for the fabric image with complex texture, the reconstructed images may exhibit regions similar to defects, thus results in the low detection accuracy.

The spectrum analysis methods transform the test image into the spectrum domain, and then utilize the energy standard of filter response to detect defects, such as Gabor transform ¹⁷ and wavelet transform. ¹⁸ These methods can capture the global characteristics of the image to detect defects, but the computational complexity is high and the detection results depends on the selected filter banks.

The above methods have achieved the ideal detection results for the fabric images with plain or twill texture, but they are lack of adaptivity, and cannot be extended to the patterned fabrics with complex texture. Recently, some researchers have proposed some methods for patterned fabric defect detection, such as the Bollinger band method (BB), ¹⁹ the wavelet-preprocessing golden image subtraction method (WGIS), ⁸ the ELO rating (ER) method, ⁷ and the TMPM method. ²⁰

The BB methods utilized the property of regularity to detect defects on dot-, box-, and star-patterned fabrics. The WGIS method utilized a golden image to perform a moving subtraction of each pixel along each row of every wavelet-pre-processed test image. The ELO rating (ER) method is a method in which the detection of fabric defects is similar to carrying out fair matches in the spirit of good sportsmanship. The TMPM method used a golden image-like approach to exploit a discrepancy measure as a fitness function to detect defects on patterned textures. These detection methods adopted wavelet transform, Gabor transform, average value, standard deviation and regular bands to characterize the fabric texture, and performs a point-to-point comparison, thus they are inherently sensitive to image noise, misalignment, and distortion.

In recent years, convolutional neural network (CNN) has made great achievements in the field of computer vision. Some researchers have adopted CNN to detect fabric defects. Ruo et al. ²¹ proposed a fabric defect detection method based on convolution neural network. It uses the pretrained network to construct the classifier, and then convolutes the image to be measured to complete the pixel level prediction. Zhang et al. ²² proposed a supervised convolutional neural network, which is trained to classify each image block and identify defects by thresholding. Although the methods based on convolutional neural network have achieved good performance, but it is a heavy workload to construct the training sets with large-scale. In addition, these CNN methods are difficult to be deployed on the production line.

For any kinds of fabric images, the background is composed of regular repeated textures, which has higher data redundancy. And the defects are sparse and salient among the background. By decomposing the feature matrix into a low-rank matrix corresponding to the background and a sparse matrix corresponding to the object, low-rank decomposition model can be well applied for the fabric defect detection.

Cao et al. ²³ proposed a low-rank decomposition model (Robust principal component analysis, RPCA) by introducing noise term to detect fabric defects. However, this method ignores the continuity of sparse pixels, which is not suitable for the continuous defect detection. Liu et al. ²⁴ proposed a fabric defect detection method based on deep convolution neural network and low rank recovery. Firstly, deep convolution neural network is used to extract fabric features and generate feature matrix, and then low rank recovery model was used to locate defects. Ji et al. ²⁵ proposed a detection method based on weighted low rank decomposition and Laplacian regularization term. By introducing Laplacian regularization term to expand the distance between background and defect area, the robustness of the model is enhanced.

Shi et al. ²⁶ proposed a decomposition model based on gradient information, which guided the low rank model to detect defects through the gradient information of the test image. Qizi et al. ²⁷ proposed a low-rank decomposition detection method based on image texture prior information. Firstly, a priori matrix is constructed according to the test image, and then the low rank model is guided by the prior matrix to detect defects. However, the performance of these methods using prior information to guide low rank models is limited by the texture prior matrix. The inaccuracy prior information will decrease the detection performance. Shi et al. ²⁸ proposed a low rank decomposition detection method based on gradient information and structure graph, which effectively solved the problem of inaccurate prior information by using structure graph algorithm. Li et al. ²⁹ proposed a novel fabric defect detection algorithm based on biological vision modeling by simulating the mechanism of biological visual perception. The detection performance is still not satisfactory because the traditional manual feature descriptors cannot efficiently characterize the fabric texture. We have listed a brief list of some methods for detecting fabric defects, as shown in Table 1.

Table 1.

Brief table of different fabric defect detection methods.

Authors	The proposed method	Specific features
Cao et al. 23	Modified RPCA with noise term and defect prior	The noise term is introduced to improve the detection accuracy
Li et al. 29	Low rank decomposition with biological vision modeling	The theory of biological vision is used to characterize the fabric
Qizi et al. 27	Low rank representation with texture prior	Prior information is used to guide the decomposition of low rank model
Ji et al. 25	Weighted low rank decomposition and Laplacian regularization	Laplacian regularization term is introduced to increase the difference between background and defect
Liu et al. 24	Deep-feature and low rank decomposition	Convolution neural network is used to extract the fabric features
Shi et al. 28	Low rank decomposition with gradient information and structured graph algorithm	The structure graph algorithm is used to improve the problem of inaccurate prior information

In order to improve the detection performance, a new powerful descriptor should be proposed to efficiently characterize the fabric texture, and make the normal regions lay in a low-rank subspace, while the defective regions are far away from the subspace. In addition, an effective low-rank decomposition model also should be developed to separate the defect from the background. However, the above methods still only use traditional manual features or deep features, which are not enough for complex texture fabric images. In addition, SVT algorithm is usually used to solve the existing low rank models, which ignores the different importance of singular values with different sizes. To solve these problems, a fabric defect detection method based on deep-handcrafted feature and weighted low rank representation is proposed in this paper.

The proposed method

A powerful feature descriptor and effective low-rank decomposition model are crucial for the low-rank based fabric defect detection method. In this paper, a fabric defect detection method based on deep-handcrafted features and weighted low rank representation is developed, and it includes three steps. Firstly, multi-layer deep features are extracted by VGG16 after transfer learning, and are cascaded with handcrafted features to effectively characterize the fabric texture. Then, an adaptive weighted low rank representation model is proposed to divide the deep-handcrafted feature matrix into low rank matrix corresponding to background and sparse matrix corresponding to defect. Finally, the defect is located by threshold segmentation algorithm. The flowchart of the proposed method is shown in Figure 2.

Figure 2.

Framework of the proposed method.

Deep-handcrafted feature extraction

A powerful feature descriptor can make the normal background lay in a low-rank subspace, while the defective regions are far away from the subspace. Therefore, feature extraction is crucial for fabric defect detection based on low rank decomposition model. The performance of traditional handcrafted descriptors depends on professional knowledge, and this designed handcrafted descriptor can be used for better characterizing some specific feature. The fabric images have complex texture, thus low-level texture feature is important for characterizing the fabric image. In this paper, we designed a handcrafted texture feature by modeling the human vision. In addition, Li et al. ¹¹ proposed that deep features contain enough high-level abstract semantic information, while low-level comparative information is insufficient. It is worth noting that deep features and handcrafted low-level features are complementary. Wang et al. ³⁰ found that deep features can be enhanced by adding handcrafted features. By combining the respective advantages of handcrafted and deep feature, the representation ability of fabric image can be enhanced.

Hand-crafted feature extraction

Weng et al. ³¹ pointed that the feature descriptors based on the mechanisms of the human visual system are suitable for characterizing complex textures, and have proposed a novel feature descriptor, denoted as distinctive efcient robust features (DERF). In this paper, we adopted this feature descriptor for better characterizing the fabric low-level texture. And it can be described as follow in detail.

Firstly, the test fabric image is divided into the image blocks $R_k$ with the size of $n_b\times n_b$ , where $k=1,2,\cdots ,N_b$ , $N_b$ is the number of image block. Then, difference of Gaussian (DoG) is used to extract the convolution gradient direction mapping of each image block as follows:

$$g_o^{\Sigma }=g_{\Sigma }*{\left(\frac{\partial R_k}{\partial o}\right)}^{+},1\le o\le H$$

(1)

where $o$ is the orientation of derivative, ${(\cdot )}^{+}=max(\cdot ,0)$ represents a nonnegative operator, $g_{\Sigma }$ is a Gaussian convolution core of scale $\Sigma$ . For each orientation map, the final Gaussian convolution gradient orientation is obtained by subtracting the large scale from the small scale of each pair of Gaussian convolution gradient orientation map:

$$G_o^{{\Sigma }_1}=g_o^{{\Sigma }_1}-g_o^{{\Sigma }_2}\mathrm{with}{\Sigma }_2>{\Sigma }_1$$

(2)

Then, the DERF descriptor is established by simulating the structure of the receptor region of P-ganglion cells in the $0^o-{15}^o$ retina. These sampling points are located on concentric circles with different radii, and their number increases exponentially with radius. $h_{\Sigma }(x,y)$ is defined as a vector generated by the value at the position $(x,y)$ with the same scale $\Sigma$ in the orientation maps of DoG convolution, and $(x,y)$ is the center of the image block:

$$h_{\Sigma }(x,y)=\left[D_1^{\Sigma }(x,y),\cdots D_i^{\Sigma }(x,y),\cdots D_H^{\Sigma }(x,y)\right]$$

(3)

where $D_i^{\Sigma }(x,y)$ represents DOG convolution patterns with different directions and the same scales. Let $s$ and $T$ be the number of layers and the sampling direction, respectively, the descriptor of each image block can be defined as the combination of $h$ vectors:

$$\begin{array}{l}{H}_k(x,y)\hfill \\ =[h_{{\Sigma }_1}(x,y),\hfill \\ h_{{\Sigma }_1}\left(l_1\left(x,y,r_1\right)\right),\cdots ,h_{{\Sigma }_1}\left(l_T\left(x,y,r_1\right)\right)\hfill \\ h_{{\Sigma }_2}\left(l_1\left(x,y,r_2\right)\right),\cdots ,h_{{\Sigma }_2}\left(l_T\left(x,y,r_2\right)\right)\hfill \\ ⋮\hfill \\ {h_{{\Sigma }_S}\left(l_T\left(x,y,r_s\right)\right),\cdots ,h_{{\Sigma }_S}\left(l_T\left(x,y,r_s\right)\right)]}^T\hfill \end{array}$$

(4)

where $l_i(x,y,r)$ represents the location with the distance $r$ from $(x,y)$ in the $i$ -th orientation, Then the handcrafted feature matrix can be obtained by stacking the features of all image blocks.

$$D_1=[H_1,H_2,\cdots ,H_{N_b}]$$

(5)

Deep feature extraction

Considering that VGG16 network has the expansibility advantage, the pre-trained VGG16 models are adopted for extracting high-level abstract semantic information in our method. Since the size of the output feature map of the convolution layer in VGG16 is not consistent, in order to improve the detection accuracy of the detects with small size, we resize the feature map to the input image. For each pixel, the deep feature is formed by concatenating the value of the feature map in the same position. Let $d_i$ be the feature vector of the $i$ -th pixel extracted from the fabric image.

$$d_i=[x_{i1},x_{i2},\cdots ,x_{il}]$$

(6)

where $i=1,2,\cdots ,N\times N$ is the image size, $x_{il}$ represents the activation obtained from the $l$ -th resized feature map of a convolution layer at the $i$ -th pixel. To construct the feature matrix, the deep feature maps are divided into image blocks of size $n_b\times n_b$ . For each image block $R_k$ , the mean value of the feature vector ${\overline{d}}_k$ of the block is taken as the feature of the image block.

$${\overline{d}}_k=\frac{{\sum }_{j=1}^{n_b\times n_b}{f}_j}{n_b\times n_b},f_j\in R_k$$

(7)

Then deep feature matrix can be obtained by stacking the features of all image blocks.

$$D_2=[{\overline{d}}_1,{\overline{d}}_2,\cdots ,{\overline{d}}_{N_b}]$$

(8)

To facilitate the fusion of depth feature and handcrafted feature, we use principal component analysis (PCA) to reduce them to the same $M$ dimension respectively, and stack them to generate the final deep-handcrafted feature.

$$D=\left[D_1,D_2\right],D\in {ℝ}^{2M\times N_b}$$

(9)

Weighted low rank representation model construction

The low rank decomposition model decomposes the feature matrix into a low rank matrix and a sparse matrix which representing the background region and the defect region of an image, respectively. Therefore, the low rank decomposition model can be used for defect detection, and has been proven applicable. Given an feature $D$ , it can be decomposed into low rank matrix and sparse matrix by the following optimization problem

$$\underset{L,E}{\min }(rank(L),\parallel E{\parallel }_l)s.t.D=L+E$$

(10)

where $L$ and $E$ are the low rank matrix and sparse matrix. $l$ -norm is used for modeling the different noise or error in $E$ , that is, the $l_0$ -norm is used for characterizing the sparse random noise; ³² $l_{2,0}$ -norm is used for modeling the special sampling noise. ³³

Optimization problem (10) is a bi-objective optimization problem, which is difficult to solve. Thus we should transform it into the following single objective optimization problem.

$$\underset{L,E}{\min }rank(L)+\lambda (\parallel E{\parallel }_{2,0})s.t.D=L+E$$

(11)

where $\parallel \cdot {\parallel }_{2,0}=\#\left\{i:\Vert {[\cdot ]}_{:,j}\Vert \ne 0\right\}$ indicates the $l_{2.0}$ -norm of a matrix, ³⁴ $\lambda >0$ is a balance factor. Since $rank(\stackrel{.}{)}$ and $l_{2.0}$ -norm are NP-hard problem, ³⁴ which have non-convex and non-smooth properties, so equation (11) is difficult to solve. To address this issues, nuclear norm and $l_{2.1}$ -norm are used to replace rank function and $l_{2.0}$ -norm, and convex substitution can be obtained using the following equation:

$$\underset{L,E}{\min }\parallel L{\parallel }_{*}+\lambda \parallel E{\parallel }_{2,1}s.t.D=L+E$$

(12)

where $\parallel \cdot {\parallel }_{2,1}={\sum }_{i=1}^n{\Vert {[\cdot ]}_{i,i}\Vert }_2$ represents the $l_{2,1}$ -norm of a matrix. In order to deal with the mixed data matrix and handle noise or corrosion more effectively, dictionary learning is introduced into low-rank decomposition model to generate a more general optimization problem model, and can be described as follows:

$$\underset{L,E}{\min }\parallel L{\parallel }_{*}+\lambda \parallel E{\parallel }_{2,1}s.t.D=AL+E$$

(13)

where $A$ is a dictionary that linearly spans the feature space. The equation (13) is a convex optimization problem which denoted as Low Rank matrix Representation (LRR). If we set $A=I$ , where $I$ is the identity matrix, equation (13) falls back to (12). Therefore, low-rank decomposition model can be regarded as a special case of low rank representation, which actually uses identity matrix as dictionary.

The problem (13) is convex and can be solved by ALM method. However, this solution method treats each singular value equally. In fact, the matrix singular values represent different information, and the larger singular value contains more information. When decomposing the feature matrix, the larger singular values corresponding to the main projection direction should be retained to the maximum extent, and it can retain the main information. For smaller singular values, more penalty should be given to reduce the influence of outliers. Therefore, the weighted nuclear norm minimization (WNNM) is proposed to shrunk the singular values of the matrix differently. The definition of WNNM is

$$\parallel \cdot {\parallel }_{w,*}=\sum _i{\left|w_i{\sigma }_i(\cdot )\right|}_1$$

(14)

where $w_i\ge$ is the nonnegative weight assigned to ${\sigma }_i(\cdot )$ , ${\sigma }_i(\cdot )$ represents the $i$ -th singular value of a matrix. In order to improve the detection performance, we integrated the WNNM the LRR model, and proposed a fabric defect detection method based on weighted low rank representation, denoted as WLRR, and it can be described as follows.

$$\underset{L,E}{\min }\parallel L{\parallel }_{w,*}+\lambda \parallel E{\parallel }_{2,1}s.t.D=AL+E$$

(15)

where $D$ is the deep-handcrafted feature matrix of the test fabric image, $L$ and $E$ are the low rank background matrix and sparse defect matrix, $\lambda >0$ is a balance factor. $A$ is a dictionary matrix that linearly spans the feature space. Because $E$ has sparse column support, feature matrix $D$ can be selected as dictionary matrix to replace A. ³⁵ Then the equation (15) can be rewritten as:

$$\underset{L,E}{\min }\parallel L{\parallel }_{w,*}+\lambda \parallel E{\parallel }_{2,1}s.t.D=DL+E$$

(16)

Optimization of WLRR

In order to facilitate the optimization solution, the variable $J$ is introduced and equation (16) is transformed into the following equivalent problem.

$$\underset{L,E}{\min }\parallel J{\parallel }_{w,*}+\lambda \parallel E{\parallel }_{2,1}s.t.D=DL+E,L=J$$

(17)

Then, the augmented Lagrangian function of equation (17) can be described as follows.

$$\begin{array}{l}L\left(J,E,D,Y_1,Y_2,\mu \right)\hfill \\ =\parallel J{\parallel }_{w,*}+\lambda \parallel E{\parallel }_{2,1}+〈Y_1,D-DL-E〉+〈Y_2,L-J〉\hfill \\ +\frac{\mu }{2}\left(\parallel D-DL-E{\parallel }_F^2+\parallel L-J{\parallel }_F^2\right)\hfill \end{array}$$

(18)

where $Y_1$ and $Y_2$ are the Lagrange multipliers, $\parallel \cdot {\parallel }_F^2$ represents the Frobenius norm of a matrix. In order to conveniently demonstrate the solution process we first introduce the following lemma. ³⁶

Let $Q$ be a given matrix. If the optimal solution to

$$\underset{W}{\min }\alpha \parallel W{\parallel }_{2,1}+\frac{1}{2}\parallel W-Q{\parallel }_F^2$$

(19)

is $W^{*}$ , then the $i$ -th column of $W^{*}$ is

$${\left[W^{*}\right]}_{:,i}=\{\begin{array}{ll}\frac{{\Vert {[Q]}_{:,i}\Vert }_2-\alpha }{{\Vert {[Q]}_{:,i}\Vert }_2}{Q}_{:,i},\hfill & \mathrm{if}{\Vert {[Q]}_{;,i}\Vert }_2>\alpha \hfill \\ 0,\hfill & \mathrm{otherwise}\mathrm{.}\hfill \end{array}$$

(20)

Based on the above lemma, equation (18) can be solved using Alternating Direction Method of Multipliers (ADMM), ³⁷ and specifically described as follows.

• (1) Updating $J$ : By fixing other variables, the optimal solution of $J$ can be obtained by minimizing the following equation (21):

$$J^{*}=\arg \underset{J}{\min }\parallel J{\parallel }_{w,*}+\frac{\mu }{2}{\Vert J-\left(L+\frac{Y_2}{\mu }\right)\Vert }_F^2$$

(21)

The optimization problem of $J$ is generally nonconvex, therefore, it cannot be solved directly. Gu et al. ³⁸ have analyzed the optimal solution of the objective function in the form of WNNM. When the weight is set to a constant, NNM is proved to be a special case of WNNM. Thus, let $P=L+Y_2/2$ and $P=US{V}^T$ , we can get the following equation.

$$J^{*}=U{S}_{w/2}(S)V^T$$

(22)

For fabric defect detection, the larger singular value of matrix contains more important components, while the traditional solution method treats all singular values equally, so the results deviate from the real solution. In this paper, we set the different weights for each singular value based on their value, and the weights are calculated as follows:

$$w_i=\sqrt{{\sigma }_i(P)}$$

(23)

where ${\sigma }_i(P)$ represents the $i$ -th singular value of $P$ . It can be seen from Figure 2 that the weighted nuclear norm can better approximate to the Rank function comparing with Nuclear norm, and can automatically assign appropriate weights to each singular value.

• (2) Updating $L$ : By fixing other variables, the optimal solution of $L$ can be obtained by minimizing the following formula:

$$L={\left(I+D^{T}D\right)}^{-1}\left(D^{T}D-D^{T}E+J+\frac{D^T{Y}_1-Y_2}{\mu }\right)$$

(24)

• (3) Updating $E$ : By fixing other variables, the optimal solution of $E$ can be obtained by minimizing the following problem:

$$E^{*}=\arg \underset{E}{\min }\frac{\lambda }{\mu }\parallel E{\parallel }_{2,1}+\frac{1}{2}{\Vert E-\left(D-DL+\frac{Y_1}{\mu }\right)\Vert }_F^2$$

(25)

The proposed WLRR can be solved by the iterative algorithm, can it can be described as Algorithm 1.

Saliency map generation

After the final sparse matrix $E^{*}$ which include the defect information is generated, the saliency score of $R_j$ block can be obtained by the $L_1$ -norm of $j$ -th column of $E^{*}$ :

$$m(I_j)=\parallel E^{*}(:,j){\parallel }_1$$

(26)

Then, the corresponding saliency map $M$ is generated according to the spatial relationship. Finally, the defect region can be obtained by binary image $\widehat{M}$ .

$$\widehat{M}(i,j)=\{\begin{array}{cc}0,& \mu -c\cdot \sigma <M(i,j)<\mu +c\cdot \sigma \\ 255,& \mathrm{otherwise}\end{array}$$

(27)

where $c$ is a constant, µ and $\sigma$ are mean and standard deviation of pixel values in the saliency map.

Experiment result

Experiments setup

Dataset

Two public fabric databases are adopted to conduct comprehensive evaluation of the proposed methods. One is TILDA fabric image dataset, ³⁹ constructed by workgroup on texture analysis of German Research Council. It is mainly composed of unpatterned fabric images. The other is from the Research Associate of Industrial Automation Research Laboratory, Department of Electrical and Electronic Engineering, Hong Kong University. It mainly includes the patterned fabric images with complex texture, such as the star-, box-, and dot-patterned fabric.

Algorithm 1

WLRR solution

Input: feature matrix $D$ ; parameter $\lambda >0$ ; Initialize: $L^0=0$ , $E^0=0$ , $J^0=0$ , $Y_1^0=0$ , $Y_2^0=0$ , ${\mu }^0=1e-4$ , ${\mu }^{\max }{=10}^5$ , $k=0$ , $\rho =1.1$ , $tol=1e-8$ while not converged do (1) Update $J^{k+1}$ by fixing the other variables: $$\begin{array}{l}{J}^{k+1}=\\ \arg \underset{J}{\min }\frac{1}{\mu }{\Vert J^k\Vert }_{w,*}+\frac{1}{2}{\Vert J^k-\left(L^k+\frac{Y_2^k}{{\mu }^k}\right)\Vert }_F^2\end{array}$$ (2) Update $L^{k+1}$ by: $$\begin{array}{l}{L}^{k+1}=(I+D^{T}D{)}^{-1}(D^{T}D-\\ D^T{E}^k+J^{k+1}+\frac{D^T{Y}_1^K-Y_2^k}{{\mu }^k})\end{array}$$ (3) Update $E^{k+1}$ by fixing the other variables: $$\begin{array}{l}{E}^{k+1}=\arg \underset{E}{\min }\frac{\lambda }{\mu }{\Vert E^k\Vert }_{2,1}+\\ \frac{1}{2}{\Vert E^k-\left(D-D{L}^{k+1}+\frac{Y_1^K}{{\mu }^k}\right)\Vert }_F^2\end{array}$$ (4) Update the Lagrange multipliers $Y_1^{k+1}$ , $Y_2^{k+1}$ , and penalty parameter ${\mu }^{k+1}$ by: $$\begin{array}{l}{Y}_1^{k+1}=Y_1^k+{\mu }^k(D-D{L}^{k+1}-E^{k+1})\hfill \\ Y_2^{k+1}=Y_2^k+{\mu }^k(L^{k+1}-J^{k+1})\hfill \\ {\mu }^{k+1}=\min ({\mu }_{\max },\rho {\mu }^k)\hfill \end{array}$$ (5) Check the convergence condition: $$\parallel D-D{L}^{k+1}-E^{k+1}{\parallel }_{\infty }<tol$$ and $$\parallel L^{k+1}-J^{k+1}{\parallel }_{\infty }<tol$$ (6)  $k=k+1$ end while Output: The optimal solution $E^{*}$

Implementation details

First, we transfer the domain-specific VGG16 to fit our fabric database by replacing the original softmax layer (i.e. whether it belongs to defect image) with 2-way output. Then, the transfer learning is carried out by stochastic gradient descent with a batch size of 16, momentum of 0.9, weight decay of 0.0001, and the learning rate is initially of 0.0001. For the WLRR model, the model parameters set to 0.001.

All parameters are kept fixed for all the experiments to demonstrate the robustness and stability of our method. The simulation is performed in MATLAB 2019a, running on a PC with an i7-7700 CPU accelerated by a HD Graphics 630 (1024MB) GPU.

Evaluation criteria

To perform a comprehensive evaluation, the statistical parameters are introduced to verify the performance, including true positive (TP), true negative (TN), false positive (FP), and false negative (FN). Based on the above parameters, evaluation criteria are defined as follows:

$$TPR=\frac{TP}{TP+FN}$$

(28)

$$FPR=\frac{FP}{FP+TN}$$

(29)

$$PPV=\frac{TP}{TP+FP}$$

(30)

$$NPV=\frac{TN}{TN+FN}$$

(31)

$$F=\frac{2\times TPR\times PPV}{TPR+PPV}$$

(32)

Moreover, F-measures (F) is also shown based on the above evaluation criteria. It should be noted that because of the lack of ground-truth in the TILDA fabric database, the above quantitative evaluation will only be conducted for the patterned fabric databases.

Experimental results and analysis

Comparison of deep features and handcrafted features

Activation of each convolution layer is used to form deep feature. We present the detection results of the first eight layers of VGG16, as presented in Figure 3. The first column is the original image; the second column to the last column are the generated saliency map using the Conv $1\_1$ to Conv $4\_1$ respectively. In addition, we present the numerical results of the first eight convolution layers to find the best convolution layer, as shown in Table 2.

Figure 3.

Comparison of the Rank, nuclear norm and weighted nuclear norm.

Table 2.

The numerical results of the first eight convolution layers.

	Conv1_1	Conv1_2	Conv2_1	Conv2_2	Conv3_1	Conv3_2	Conv3_3	Conv4_1
TPR	0.15	0.17	0.2	0.36	0.36	0.24	0.28	0.19
FPR	0.08	0.02	0.05	0.07	0.09	0.07	0.08	0.09
PPV	0.08	0.15	0.11	0.17	0.16	0.14	0.15	0.09
NPV	0.96	0.98	0.98	0.98	0.97	0.96	0.96	0.96
F	0.11	0.16	0.14	0.23	0.22	0.18	0.19	0.12

Bold indicates the best performance.

As can be seen from Figure 4 and Table 2, the deepest and shallowest convolution layers produce worse results, while the middle layer achieves better detection results. Among them, the Conv $2\_2$ achieves the best performance, so Conv $2\_2$ in VGG16 is selected to fuse with handcrafted features in our method.

Figure 4.

The saliency maps of different convolution layers.

In order to perform a quantitative evaluation, we list the detection results of handcrafted features and deep-handcrafted features to prove the effectiveness of the proposed fusion strategy, as shown in Figure 5. And the numerical results are shown in Table 3. According to the quantitative and qualitative experiments, we can conclude that the deep features and handcrafted features complement each other, and the detection results are better than the single feature.

Figure 5.

Detection results for the unpattern images and pattern fabric images with three configurations. The first row is the original image, detection results of LRR, PLRR, WLRR are listed from the second row to the fourth row.

Table 3.

The numerical results of the deep features, handcrafted features, and deep-handcrafted features.

	F	TPR	FPR	PPV	NPV
Deep feature	0.23	0.36	0.07	0.17	0.98
Handcrafted features	0.24	0.29	0.07	0.21	0.95
Deep-handcrafted features	0.48	0.54	0.03	0.42	0.98

Bold indicates the best performance.

Comparisons of saliency inference models

In section III, a saliency inference model WLRR is proposed to generate the defect saliency map. To validate its effectiveness, we compare our proposed model with the other two models: (1) original LRR (i.e. RPCA when dictionary matrix is identity matrix); (2) LRR based on other weighted nuclear norms (Schatten p norm, see Ref. ⁴⁰ ). The results are shown in Figure 6. The first row is the original image, and the last three lines are saliency maps generated by LRR, PLRR, and WLRR.

Figure 6.

The detection results of handcrafted features and deep-handcrafted features.

In our method, the weight operator can preserve more defects information in sparse matrix, thus can get the accurate results. From Figure 6, espically for the sixth image, we can see that WLRR retain more defect profile information comparing with LRR and PLRR.

To further prove the efficiency of the proposed methods, the numerical results of the three methods in the pattern fabric image are shown in Table 4. We can conclude that WLRR performs better than the other two. Both qualitative and quantitative experiments confirm that the effectiveness of WLRR is more suitable for the fabric defect detection.

Table 4.

The numerical results of LRR, PLRR, WLRR.

	F	TPR	FPR	PPV	NPV
LRR	0.35	0.43	0.04	0.3	0.98
PLRR	0.44	0.41	0.03	0.49	0.96
WLRR	0.48	0.54	0.04	0.42	0.98

Bold indicates the best performance.

Comparisons with the state of the arts

In this section, we compare the detection results of our method with the state-of-the art, including GABOR, ⁴¹ LSF-GSA, ⁴² ESP, ⁴³ and SOMC. ⁴⁴ All of the saliency maps are shown in Figures 7 to 9, which show comparisons of box-patterned fabrics, dot-patterned fabrics and star-patterned fabrics. The first row is the original fabric image, and the second to the sixth rows are saliency maps generated by the GABOR, LSF-GSA, ESP, SOMC methods, and our proposed method. The seventh row is the segmentation results generated by our method and the last row is the ground-truth images.

Figure 7.

Detection results for box-patterned fabric images: The defect types are (a) broken end, (b) hole, (c) netting multiple, (d) thick bar, and (e) thin bar.

Figure 8.

Detection results for star-patterned fabric images: The defect types are (a) broken end, (b) hole, (c) netting multiple, (d) thick bar, and (e) thin bar.

Figure 9.

Detection results for dot-patterned fabric images: The defect types are (a) broken end, (b) hole, (c) knots, (d) netting multiple, (e) thick bar, and (f) thin bar.

It can be observed that there are more discontinuous regions in Gabor method, and there are many missing results. LSF-GSA combines local texture features with global analysis to generate saliency map, but the detection effect is poor and there are many noises. The ESP method has poor detection effect, especially in the detection of defects similar to the background, which shows that the method is not suitable for the detection of complex pattern fabrics. SOMC method can effectively detect the location and contour of defects, but its detection effect on dot fabric is worse than the other two fabrics. In addition, the performance of this method is poor for the defect with large size. Our method can not only detect the location of defect area, but also describe the defect contour of fabric image. The binary image obtained by segmentation is similar to the true value image. In addition, as shown in Figures 7(d) and 8(a), the proposed method can detect fabric images with large defects more effectively than other methods.

In the quantitative evaluation, we give the evaluation results of the above methods on box, dot, and star pattern fabric images. The best performing data will be shown in bold. Since the saliency map generated by ESP method are binary graphs, they are not evaluated numerically. From Table 5, we can see that in most cases, the performance of our proposed method on three patterned fabric datasets is better than the existing methods. In summary, the qualitative and quantitative experiments prove the superiority of our method.

Table 5.

Average numerical detection results of different patterns of fabric images.

Pattern	Method	F	TPR	FPR	PPV	NPV
Box	GABOR 41	0.34	0.29	0.02	0.4	0.97
	LSF-GSA 42	0.23	0.24	0.06	0.23	0.94
	SOMC 44	0.39	0.51	0.02	0.32	0.99
	WLRR (OURS)	0.42	0.52	0.03	0.35	0.99
Star	GABOR 41	0.33	0.33	0.02	0.33	0.98
	LSF-GSA 42	0.28	0.28	0.07	0.28	0.93
	SOMC 44	0.47	0.66	0.02	0.36	0.99
	WLRR (OURS)	0.52	0.71	0.05	0.41	0.98
Dot	GABOR 41	0.32	0.22	0.02	0.54	0.94
	LSF-GSA 42	0.31	0.22	0.04	0.54	0.85
	SOMC 44	0.49	0.45	0.03	0.52	0.96
	WLRR (OURS)	0.41	0.33	0.02	0.54	0.95

Bold indicates the best performance.

Conclusion

In this paper, we proposed a novel fabric defect detection method on deep-handcrafted feature and WLRR. Based on the fact that manual low-level features and deep features can complement each other, the deep features extracted from VGG16 are fused with handcrafted features to improve the representation ability of fabric texture. Then, the fabric image is decomposed into background part and sparse defect part by LRR. At the same time, the weighted kernel norm is used to regularize the background matrix, so that the singular value of the matrix can be shrunk differently to retain the most important information. We also compare the performance of the proposed method with the previous methods (e.g. Gabor, GSA, ESP, SOMC). The results of qualitative and quantitative experiments show that our method is more effective than the state-of-the-art. In addition, the proposed method provides a new solution for the surface anomaly detection of other industrial products.

Footnotes

Acknowledgements

The authors would like to thank Dr. Henry Y.T. Ngan, Industrial Automation Research Laboratory, Department of Electrical and Electronic Engineering, The University of Hong Kong, for providing the database of patterned fabric images.

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by NSFC (No. U1804157, 61772576, No. 62072489), Henan science and technology innovation team (CXTD2017091), IRTSTHN (21IRTSTHN013), Program for Interdisciplinary Direction Team in Zhongyuan University of Technology. ZhongYuan Science and Technology Innovation Leading Talent Program (214200510013).

ORCID iD

Ban Jiang