Fabric Defect Detection Using Deep Convolution Neural Network

Abstract

Abstract

Fabric defect detection plays an increasingly important role in the industrial automation application for fabric production, but how to detect defects rapidly and accurately is still challenging. In this study, we propose a powerful fabric defect detection method using a hybrid of convolutional neural network (CNN) and variational autoencoder (VAE). The convolutional layers are used for extracting fabric image pattern features and the variational autoencoder is used for modeling the latent characteristics and inferring a reconstruction. The defect positions can be detected by the differences between the original image and the reconstruction image. The proposed method is validated on public patterned fabric datasets. The experimental results demonstrate that the proposed model can achieve outstanding performance in both image level and pixel level defect detection.

Introduction

As a hot topic in automation, fabric defect detection is a necessary and essential step for quality control in the textile manufacturing industry. In recent years, this automatic fabric defect detection method has been developed. The traditional method heavily relies on manual inspection to detect fabric defects, which can help the working pipeline to repair minor defects immediately. However, the efficiency of manual detection will gradually decrease with increased manual working time. Therefore, it is necessary to develop an automatic inspection system for fabric defects to improve fabric quality and reduce labor costs and errors. Since most textile industries are bulk producers of textiles, the probability of the appearance of the defects will be high. In that case, classification based on deterministic patterns can also be accomplished through the design of an automated system based on computer vision. High complexity data obtained from real-time fabric design has increased complexity due to high-order variables, high diversity data, and multi-dimensional data sets formed by images.

Most of the traditional fabric defect detection methods are based on auto-correlation function (AF), ¹ local binary pattern (LBP), ² Fourier transform (FT), ³ wavelet transform (WT), ⁴ and neural network^5,6 methods. These methods are used to detect defects at image level, so it is difficult to locate defects accurately.

Deep learning (DL)^7–9 methods have a highly optimized weight and effectively extract the required functions. By using advanced algorithms such as a convolution neural network (CNN), the DL method can be applied to defect detection.^10,11,13 Compared with traditional works, these DL models have achieved excellent performance in the application of fabric defect detection. Autoencoders (AEs) are simple DL networks that transform inputs into outputs with the minimum possible error. AE-based fabric defect detection methods have been well explored and achieved good performance.^10–12 For example, Seker et al. proposed DL-based method for fabric defect detection using the AE model. ¹¹ Furthermore, Li et al. proposed a stacked denoising autoencoder (SDAE)-based method for deformable patterned fabric defect detection, where the Fisher criterion was introduced into the model optimization process to improve discrimination between defective and defect-free patches. ¹⁰ Benefiting from the unsupervised characteristic of the auto-encoder, those AE-based fabric defect detection methods achieve better performance than the traditional methods, and are easy to implement in practice.

In this study, we focus on the design and development of a computer vision application for fabric defect detection and classification using a hybrid model based on deep CNN and a variational autoencoder (VAE), which is named as HVAE. HVAE has shown good performance on data representation by providing a probabilistic manner for describing an observation in latent space.^14,15 Therefore, we use convolutional layers to extract fabric image pattern features and the HVAE to model the latent characteristics. Finally, the defect positions can be detected by the differences between the original image and the reconstruction after postprocessing by thresholding and morphology filtering. The proposed model is evaluated on a public patterned texture fabric data-set. The experimental results demonstrate that the proposed model is outstanding on patterned texture fabric defect detection at both image and pixel levels.

Generally, the contributions of our paper can be summarized as follows:

• •

  We propose a hybrid model based on a deep CNN and an HVAE that detects fabric defects more accurately. Instead of using the original pixel value as input by the existing AE-based methods, we use CNNs for image texture feature extraction and the HVAE for features representation.

• •

  Different than most current CNN-based fabric defect detection methods that need a large scale labeled defect database, only defect-free samples are required for the training by the proposed model. This makes it easy to use in practice due to fabric defect samples being labor-consuming to collect and mark.

• •

  We validated the proposed model on a public fabric data-set and performed quantitative and qualitative analysis to demonstrate its effectiveness.

The rest of this paper is structured as follows. Related works on traditional fabric defect detection methods and fabric detection methods based on DL are introduced. The proposed model for fabric defect detection is then demonstrated. The datasets used in our study are introduced. The first dataset is used for defect classification of five defect categories and the section dataset is used to distinguish between defective and non-defective fabric images. This is followed by experimental results and the conclusion.

Related Work

Fabric defect detection has been a popular research topic for many years and plenty of studies related to fabric defect detection have been reported.^16–18 The previously limited research for fabric defect detection is reviewed as follows.

Fabric Defect Detection Methods

The most common methods for fabric defect detection can be included in four categories, including low level feature-based, frequency-based, model-based, and learning-based methods.

Low level feature-based methods, researched in early years, distinguished defects by using feature analysis of standard textile texture patterns or structures based on the spatial distribution of gray values, ¹⁹ such as using morphological operations, ²⁰ bi-level thresholding, ²¹ and edge detection. ²² Obviously, those methods easily fail when discriminating blurry defects that do not change the image, in color or detecting defects in fabrics of irregular texture.

Frequency-based methods extract fabric defects by detecting abnormal values using Fourier transform, wavelet transform, Gabor transform, and filtering approaches ²³ For example, Li and Zhang developed an embedded machine vision system using Gabor filters and Pulse Coupled Neural Network (PCNN) that can identify defects of warp-knitted fabrics automatically. ²⁴ However, the selection of appropriate parameters for both wavelets transform coefficients and Gabor filters then becomes the most challenging task in defect detection.

Model-based methods detect texture defects based on the construction of an image model.^25–27 Regression-based ²⁸ and Gauss Markov Random Field ²⁹ models are most commonly-used in model-based defect detection methods. This technique outperforms only when fabric images have surface changes due to defects such as yarn breakage and needle breakage. Even though it consumes less computational time, no reliable studies have been made in recent years.

Learning-based methods use labelled samples to train classifiers that distinguish between defective and non-defective samples. There are fabric defect detection studies made using classifiers such as the Support Vector Machines (SVM), ³⁰ feedback Artificial Neural Network (ANN), ³¹ and the Bayes classifier (BC) ⁸ to learn signatures of defected and non-defected classes. But most of those pattern classification methods need a large variety of data. Moreover, once the algorithm is trained for one particular data set, the network structure cannot be modified, which is inconvenient for practical applications.

Fabric Defect Detection with DL

Taking advantage of large-scale data and high computing power, DL^6,7 methods are able to learn richer hierarchical features from training samples and exhibit remarkable performance in a wide range of applications, such as action recognition, ³² traffic flow prediction, ⁵ and face recognition. ³³

By using an advanced algorithm such as CNNs,^34,35 DL has also been used to address the problem of fabric defect detection and classification.^{10,11,13,18,36} Seker et al. proposed deep learning-based method for fabric defect detection using an AE model. ¹¹ Li et al. proposed an SDAE-based method for deformable patterned fabric defect detection, where the Fisher criterion was introduced into the model optimizing process to improve discrimination between defective and defect-free patches. ¹⁰ To overcome the weakness of the single scale neural network, Mei et al. proposed a multiscale convolutional denoising AE approach that simultaneously detects and localizes defects in fabrics by combining multiple ConvNet models with Gaussian pyramid decomposition. ¹² Zhang et al. built a convolutional network (ConvNet) model from the well-known AlexNet for yarn-dyed fabric defect classification. ³⁶ Furthermore, Wang et al. built a dual ConvNets architecture for fabric defect detection, where the first ConvNet was used for fabric category classification, and the second ConvNet was category specific and used for defective regions detection. ¹³ Although there are several notable works of applying deep neural networks on detecting defects, deeper CNN methods are still under exploration for higher performance of fabric defect detection.

The VAE outperformed the existing AE methods on data representation by providing a probabilistic manner for describing an observation in latent space.^14,15 In this study, we propose a hybrid model based on a deep CNN and a VAE. Differently from the VAE, we use convolutional layers for fabric image features extraction instead of using the original value of the image as the input of the VAE directly.

Proposed HVAE Model

In this section, details of the HVAE method are presented. Our proposed method is composed of a CNN and a VAE. In the training step, normal defect-free images are segmented into patches and then fed to the model for fine-tuning the parameters. In the testing step, the trained model is used to slide on the original image and rebuild a new image. The differences between the original image and rebuilt image can be considered for defect detection after thresholding and morphology processing.

Review of Variational Autoencoders

The VAE^14,15 is one of the most popular frameworks for image generation. Following the common techniques for latent variable models, the main idea of VAE is using a probabilistic encoding network to map a latent variable zto an observed input x and using an inference network to perform the reverse mapping from x to z. The two networks form a computational pipeline that resembles an unsupervised autoencoder.

In the computing process, we use the probabilistic encoding network q_ϕ(z|x) as the approximation to the posterior of the inference network p_θ(x, z). Let the prior over latent variable be the centered isotropic multivariate Gaussian p_θ(z) and p_θ(x|z) be a multivariate Gaussian or Bernoulli distribution. The prior lacks parameters, but the true posterior p_θ(z|x) is trackable. Therefore, the variational approximate posterior can be constructed as shown in Eq. 1.

$$\begin{array}{l}{p}_{\theta }(z)=\mathbb{N}(z;0,1)\text{ and }p\theta (x\mid z)\\ \log q_{\varphi }\left(z\mid x_i\right)=\log \mathbb{N}\left(z;{\mu }_i,{\sigma }_i^2\right)\end{array}$$

Eq. 1

μ _i and σ² _i , which denote the mean and standard of the approximate posterior, are the outputs of the encoding functions of data point x ⁱ .

The posterior z _i,l ∼ q_φ(z|x _i ) can be sampled by z _i,l = g_φ(x_l, ε _l ) = μ _i + σ _i ⊙ ε _l , where ε _l ∼ ℕ(0,1). Using the Kullback-Leibler divergence (KL-divergence) that can be computed and differentiated without estimation, the final estimator can be constructed as Eq. 2.

$$\Lambda \left(\theta ,\varphi ;x_i\right)\simeq \frac{1}{2}{\displaystyle \sum _{j-1}^J\left(1+\log \left({\left({\sigma }_{i,j}\right)}^2\right)-{\left({\mu }_{i,j}\right)}^2-{\left({\sigma }_{i,j}\right)}^2\right)}+\frac{1}{L}{\Sigma }_i^{L{\Sigma }_{{\theta }_{i_{i,l}}}}\log$$

Eq. 2

HVAE Model Network

To detect the defect areas in the fabric image correctly, we designed our model with a hybrid of CNN and VAE. Therefore, the proposed network architecture includes two major parts: the Conv-DeConv part used for extracting features from images, and the encoder-decoder part used for reconstructing the image. As mentioned above, the CNN network is good at image processing.

As shown in Fig. 1, the HVAE network model consists of 15 layers. The architecture can be denoted by Conv(32, 3, 3, 2) - Conv(64, 3, 3, 2) - Conv(128, 3, 3, 2) - Flat - FC(512) -FC(20) + FC(20) - FC(512) - FC(2048) - Reshape (128, 4, 4) - DeConv(128, 3, 3, 2) - DeConv(64, 3, 3, 2) - DeConv(32, 3, 3, 2) - DeConv(128, 3, 3, 1). Here, Conv(n, 3, 3, k) represents a convolutional layer with n filter of kernel size 3 3 and strike of k Flat represents flatting the 2D feature map to a 1D feature map. FC(n) represents a fully connected layer with n neurons. Reshape (128, 4, 4) represents reshaping the 1D feature map to a 128 4 4 feature map. DeConv (n, 3, 3, k) represents a deconvolutional layer with n filter of kernel size 3 3 and strike of k To reduce the computation cost of the model, we use a downscale strategy of 2-strike on the convolutional operation. We also apply a dropout strategy of 0.4 on the fully connection between layers to prevent the model from overfitting.

Fig. 1

The architecture of our proposed HVAE network.

To generate samples from parameter distribution, we apply a reparameterization trick that is useful for model proposed by Kingma et al. ¹⁴ Let the hidden variable z ∼ p(z|x) = ℕ(μ, σ²), then the reparameterization can be expressed as z = μ + σε, where ε is an auxiliary noise variable ε∼ ℕ(0,1). Therefore, in the practical implementation of reparameterization, fully connected layers are applied to build the relationship between μ, σ, and z.

Model Training

The purpose of model training is to learn the distributiion of the defect-free image patches. The model can generate a reconstruction image similar to the input image. In the training process, no defect fabric image is needed.

Defining the Loss Function

Suppose that there exists some hiddien variable z that genereates an observation x. We can infer the characteristics of z from x with _p (z|x) using Eqs. 3 and 4.

$$p(z\mid x)=\frac{p(x\mid z)p(z)}{p(x)}$$

Eq. 3

$$p(x)={\displaystyle \int p}(x\mid z)p(z)dz$$

Eq. 4

However, p(x) turns out to be an intractable distribution. Therefore, we use another distribution q (z|x), which can be defined as a tractable distribution, to approximate p(z|x). If we can define the parameters of q(z|x) such that it is very similar to p(z|x), we can use it to perform approximate inference of the intractable distribution. The KL-divergence is a measure of difference between two probability distributions. ²⁵

Thus, if we want to make the distribution of p(z|x) similar to q(z|x), we could minimize the KL-divergence between them, as expressed in Eq. 5.

$$\min KL(q(z\mid x)\Vert p(z\mid x))$$

Eq. 5

Alternatively, Eq. 5 can be calculated as maximizing Eq. 6.

$$E_{q(z\mid x)}\log p(x\mid z)-KL(q(z\mid x)\Vert p(z))$$

Eq. 6

Therefore, the loss function for the network will consist of two terms, which is defined in Eq. 7.

$$\Lambda \left(x,x^{\prime }\right)-{\displaystyle {\sum }_{j}KL}\left(q_j(z\mid x)\Vert p(z)\right)$$

Eq. 7

Here, the first term is used to penalize reconstruction error, which is maximizing the reconstruction likelihood. The second term is used to encourage the learned distribution q (z |x) to be similar to the true prior distribution p (z), which is assumed to follow a unit Gaussian distribution, for each dimension j of the latent space.

Training Process

As shown in Fig. 2, the first step of training the proposed model is extracting patches from defect-free fabric images, which are used to create the training datasets. Then, the training datasets are fed into the model, which output the reconstruction patches and the latent state distributions, that is the means and variances. As the loss definition above, the general loss includes two part: one is to evaluate the similarity between the input patches and the reconstruction patches; the other part is the KL-divergence of the latent distributions.

Fig. 2

Flowchart of training the proposed model.

In our experiment, the patch size is set to w × h. To generate a larger dataset for training and increase the robustness of the training process, we used a sliding window of w × h with strike of s. Therefore, for a fabric image with size W × H, the count of patches generated from each defect free fabric image can be calculated using Eq. 8.

$$\text{N}=\lceil [(\text{W}-\text{w})/\text{s}+1]\times [(\text{H}-\text{h})/\text{s}+1]\rceil$$

Eq. 8

Model Testing

Model testing is used to evaluate the performance of the trained model. The process includes patch extraction, image reconstruction, and locating the defects, as shown in Fig. 3.

Fig. 3

Flowchart of testing the proposed model.

Image Reconstruction

The image reconstruction process can be described as follows. First, the candidate images are segmented into patches with the same size in the training process. But different fro m the training process, the slipping window to generate patches for testing is set to w × h with strike as thee window size. Therefore, each candidate image after patch extraction can be expressed as [N_p,C,w,h], where $N_p\lfloor \left(W/w\right)\times \left(H/h\right)\rfloor$ , where refers to the number of patch samples and C refers to the number of image channels. Second, the patches are fed into the trained HVAE model and generate the reconstructed patches. Finally, the reconstruction image is orderly combined with the reconstructed patches.

Defects Location

When the reconstructed image is generated, the difference image between candidate image and reconstructed image can be calculated. After that, the defect can be located by thresholding and morphology filtering on the difference image. The performance of defect location not only depends on the quality of the reconstructed image, but also depends on the post-process methods, such as thresholding and morphology filtering. In this study, we segment the defect using Eq. 9.

$${\mathrm{defect}}_{(x,y)}=\{\begin{array}{l}1,{\mathrm{img}}_{(x,y)}\ge T\hfill \\ 0,{\mathrm{img}}_{(x,y)}<T\hfill \end{array}$$

Eq. 9

Here, (x,y) denotes the position of the candidate pixel, defect_(x,y) denotes the defect detection mask, img _(x,y) denotes the difference image, and T denotes the given thresholding value.

Dataset

To demonstrate the performance of our method, we use a patterned texture fabric for the experiment. The fabrics (box-patterned fabric, dot-patterned fabric, and star-patterned fabric) in the patterned fabric dataset was collected by Henry Y.T. Ngan and Grantham K.H. Pang from the Industrial Automation Research Laboratory, Dept. of Electrical an d Electronic Engineering, at The University of Hong Kong, which consists of three different textures with five types of defects. ¹⁶ There are a total of 25 defect images and 30 defect-free images in each type of texture. The size of the fabric image is 256*256. Fig. 4 shows some typical defect images.

Fig. 4

The typical defect samples of patterned texture fabric images. The defects include netting-multiple, broken-end, hole, knot, thick-bar, and thin-bar.

Experiments and Discussion

We evaluate the performance of the proposed model on both image level and pixel level detection accuracies.

Implementation and Evaluation Criteria

The processor is Intel (R) Core (TM) i5 - 8300H CPU @ 2.30 GHz, and RAM 8.00 GB. The graphics card type is NVIDIA GeForce GTX 1060, with a di splay memory of 6.00 GB. TensorFlow 2.0 is used for implementation.

For evaluation, we adopted the criteria used in previous work,^11,24 that is, evaluate the model performance from two aspects: image-level and pixel-level metrics.

On the imaage-level, the classification performance can be evaluated by detection rate (DR), false alarm rate (FAR), and accuracy (Acc), which are widely used metrics for defect detection. These metrics are define d in Eqs. 10–12.

$$DR=TP/N_{defect}\times 100\%$$

Eq. 10

$$FAR=FP/N_{defect-free}\times 100\%$$

Eq. 11

$$Acc=(TP+TN)/\left(N_{defect}+N_{normal}\right)\times 100\%$$

Eq. 12

TP denotes the number of true predictions of defective images, FP denotes the number of falsely predictions to defective images, and TN denotes the number of true predictions of defect-free images. N _defect denotes the number of defect images, an d N _normal , denotes the number of defect-free images.

On the pixel-level, the classification model can be evaluated by precision, recall, and F1 score. Precision is a valid choice of evaluation metric when we want to be very sure of our prediction, while recall is a valid choice of evaluation metric when we want to capture as many positives as possible. The F1 score is a number between 0 and 1 and is the harmonic mean of precision and recall. These metrics are defined in Eqs. 13–15.

$$\text{ Precision }=\left(T{P}_p\right)/\left(T{P}_p+F{P}_p\right)\times 100\%$$

Eq. 13

$$\text{ Recall }=\left(T{P}_p\right)/\left(T{P}_p+F{N}_p\right)\times 100\%$$

Eq. 14

$$F_1-\text{ Score }=2\times \frac{\text{ Precision }\times \text{ Recall }}{\text{ Precision }+\text{ Recall }}\times 100\%$$

Eq. 15

Here, TP _p denotes the proportion of pixels correctly segmented into the defective area, while FP _p denotes the proportion of pixels falsely segmented into the defective area. FN_p denotes the proportion of pixels falsely segmented into the defective-free area.

Accuracy of Patterned Fabric Database

Our proposed HVAE model is capable of learning the pattern characteristics of the fabric image, which is the essential step for fabric defect detection. Samples of the reconstruction image and defect location result are shown in Fig. 5.

Fig. 5

Samples of defect detection by the proposed model on (a) box-patterned, (b) dot-patterned, and (c) star-patterned fabric image.

On the pixel-level evaluation, the experiment contains 26 defect images and 30 defect-free images on a box-patterned texture (BOX), 30 defect images and 30 defect-free images on a dot-patterned texture (DOT), and 25 defect images and 25 defect-free images on a star-patterned texture (STAR). We use a coarse, but effective, method for determining whether the candidate image is defective or not by calculating the number of defective pixels on the resulting matrix. If the number of the defective pixels is larger than a given threshold, we consider the candidate image to contain defects. Otherwise, the defective pixels are considered as noise. In practice, we set the threshold as 150.

Table I shows the results of the pixel-level defect detection. On average, our model achieves classification accuracies of 86% on box-patterned fabric images, 98% on dot-patterned fabric images, and 98% on star-patterned fabric images.

Table I.

Classification Results of Defect Detection on Image Level

Metrics	DR	FAR	Acc
BOX	80% (21/26)	7% (2/30)	86% (48/56)
DOT	97% (29/30)	0% (0/30)	98% (59/60)
STAR	97% (24/25)	0% (0/25)	98% (49/50)

On the pixel-level evaluation, only the defect candidate image was used for testing. The classification results are shown in Table II. Our method achieves an average F1 score of 46% on box-patterned images, 53% on dot-patterned images, and 38% on star-patterned images.

Table II.

Classification Results of Defect Detection on Pixel Level

Metrics	Precision	Recall	F1 Score
BOX	39%	55%	46%
DOT	56%	61%	53%
STAR	45%	39%	38%

Compared to the regular VAE-based method, we use additional CNN layers for fabric texture pattern feature extraction, which is considered more suitable for image feature extraction.^10,11 From the evaluation results on the patterned texture fabric image dataset, we found that our model achieves outstanding performance on fabric image reconstruction. Based on the difference between reconstruction and candidate images, our model achieves an average classification accuracy of 94% (86% on box-patterned fabric images, 98% on dot-patterned fabric images, and 98% on star-patterned fabric images) on detecting whether the candidate image is defective. For defect location, we adopt a coarse method that uses threshold and morphology filtering. This method is unsupervised and efficient, but limited in the performance of defect location. That is why the pixel-level defect classification only achieves an F1 score of 46% on box-patterned fabric images, 53% on dot-patterned fabric images, and 38% on star-patterned fabric images. Therefore, after image reconstruction, the defect location method is also essential for defect detection.

Conclusions

In this study, we propose a powerful fabric defect detection method using a hybrid of a convolutional neural network (CNN) and a variational autoencoder (VAE). Three convolutional layers are used for extracting fabric image pattern features, and the VAE is used for modeling the latent characteristics and inferring a reconstruction. The defect positions can be detected by the differences between the original and the reconstruction images. The proposed method is validated on a public patterned fabric dataset. The experimental results demonstrate that our proposed model achieved outstanding performance in terms of both accuracy and robustness. Generally, our method can improve efficiency, shortening the time of model training, and obtain an outstanding accuracy on detecting various fabric defect images.

In future work, we will focus on two main directions of research. One direction involves the defect segmentation based on a full convolution network. The other direction is to develop a ConvNets model based on second-order pooling for fabric classification, which could be used before defect detection.