Sage Journals: Discover world-class research

Abstract

Image processing has been employed in a variety of fields since the advent of image processing techniques. One of these fields is textiles. The existence of any defect in a fabric is one of the most important factors affecting the quality of the fabric. There are many types of fabric defects that can occur for various reasons. It is critical to figure out what caused the defect and fix it so that it does not occur again. Automation of fabric defect detection has recently attracted a great deal of interest in view of the development in artificial intelligence technology to be able to discover defects with a high degree of success and to limit the harm to the manufacturer. This study focuses on analyzing different feature extraction methods and different classifiers and discussing the advantages/disadvantages of the combinations and, unlike other studies, using feature fusion for feature extraction. Different cases have been created that handle fabric datasets from different angles and apply different methods of feature extraction (convolution neural network, minimum relevance and maximum redundancy) and classification (ensemble learning (EL), k-nearest neighbor, support vector machine (SVM)) for separating defected and un-defected patterned and un-patterned fabrics. ResNet18 is the convolution neural network-based model with the highest performance in feature extraction, while EL and the SVM allow us to achieve close and highly successful results in classification. When feature fusion is used, ResNet18 & GoogLeNet & SVM is the most successful combination compared to the others (94.66%).

Keywords

Fabric defect classification feature selection feature fusion machine learning deep learning

Cloth imperfections alter how fabric appears. Numerous negative effects, including consumer unhappiness and financial loss, result from this. A fabric's price drops by 45–65% if there is a defect in it.¹ Intervention as soon as a defect appears is essential for stopping fabric defects. Traditionally, human eyesight has been used to identify fabric defects. Despite being a ubiquitous sort of control, it has several drawbacks, such as a loss of human power, consumption of time, low success rate, higher cost, and high rate of error. Given the drawbacks, traditional defect control cannot fully live up to expectations. Numerous experiments on autonomous fabric management have been done over the past 20 years to prevent these drawbacks.²

To automate systems run by human power, defect detection systems have been created. Automatic detection has several benefits, including lessening the loss of human power, cutting down on the time and labor expense needed for control, providing more precise results, and recording the detection process to catch future defects.³ Some of the automated systems in this field are merely designed to identify problems; others classify them after doing so. Studies to categorize defects only do so for certain defect types due to the vast array of defect types that might exist in textiles.

The objectives of this study can be listed as follows:
to perform fabric defect control, which is traditionally done using human power, with a high accuracy rate via computer vision;

to design a lower-cost automated system with higher success, instead of a costly control process with 70% success in the best case;

to save the manufacturer, who wants to perform the control process automatically, from foreign dependency and to create a much more affordable domestic system.

This study consists of six sections. The rest of this study is as follows. In the second section, a detailed literature review is included. The third section deals with the materials and the methods used in this study. In the fourth section, the proposed system is included. The fifth section includes experimental results. The conclusion and future works are given in the sixth section.

Related works

There are several types of automated fabric defect detection and classification systems recommended in the literature. It is seen that the artificial neural network (ANN) is used in many studies. Kuo and Lee,⁴ Huang and Chen,⁵ Kumar,⁶ Behera and Mani,⁷ Büyükkabasakal,⁸ Çelik et al.,⁹ Jmali et al.,¹⁰ and Rebhi et al.¹¹ propose ANN-based studies. Kuo and Lee⁴ employ a feedback neural network with three defect characteristics retrieved (maximum length, maximum breadth, and defect gray level). Huang and Chen⁵ create a fuzzy logic-based neural network. Kumar⁶ identifies the defects in twill and plain-woven textiles using a feedforward neural network. Behera and Mani⁷ use the discrete cosine transform (DCT) in conjunction with a back propagation neural network. Principal component analysis (PCA) has been used by Büyükkabasakal⁸ to create a recognition system based on ANNs. Feed-forward neural networks have been employed by Çelik et al.⁹ after filtering and morphological operators. The five characteristics (vertical energy, horizontal energy, diagonal energy, energy mean, and energy standard deviation) retrieved from the DCT of H-images (homogeneity images derived from input images) are used by Rebhi et al.¹¹ In recent years, the popularity of convolutional neural networks (CNNs) has increased and the CNN has been used not only in this field, but also in many studies in different fields. Wei et al.¹² suggest a new approach that combines compressive sensing with the convolutional neural network (CS-CNN). Şeker et al.¹³ employed an auto encoder to improve feature extraction performance in their system.

In this study, the public part of the Tilda dataset is used. The studies that will be mentioned next are among the studies using the Tilda dataset. While some of these studies distinguish the images as defected/un-defected, some perform classification according to defect types. Gabor wavelets,^14–16 PCA,^14,15 independent component analysis (ICA),¹⁷ the local binary pattern (LBP),^18,19 the gray level co-occurrence matrix (GLCM),^18,19 support vector machines (SVMs),^19–23 and k-nearest neighbors (KNN)^21,22 are used in the majority of fabric defect detection studies. The methods of the histogram, co-occurrence matrix, and shape descriptor are used for feature extraction, while the SVM is used for classification in the study of Murino et al.²⁰ Images are divided into sub-windows in the study of Basibuyuk et al.²⁴ Autoregressive coefficients are calculated using randomly selected un-defected images. Jing et al.²⁵ use the distance matching function to determine the frequency of repetition of the pattern in patterned fabric, and two properties calculated using the regular band are determined as fabric defects. To detect defects on fabrics with complicated textures, Qu et al.²⁶ develop a system based on a dual-sale over-complete dictionary. The disadvantage makes it unsuitable for online fabric assessment. Kure et al.¹⁶ investigate homogeneity in fabric images. Local neighborhood analysis is used to measure homogeneity. After partitioning the images into blocks, feature vectors extracted from each block are used in a regression-based method, which is named PG-LSR, in the study of Cao et al.²⁷ The study of Kaynar et al.¹⁸ is the first study using an ANN on this dataset. Liu et al.²⁸ use the ELM (extreme learning machine) method after extracting the features from segmented defects in fabrics. Jing et al.²⁹ use a CNN after division of the images into patches. Jeyeraj and Nadar³⁰ use a transfer learning-based CNN algorithm called AlexNet. The effect of different block sizes in the study of Cuifang et al.²³ has been examined using a pyramid histogram of edge orientation gradients (PHOG). It is seen that the detection rate increases as the size increases.

It is feasible to expand the number of prior studies mentioned above that are comparable. When looking at the studies using the Tilda dataset, the analyzed studies attempted to identify patterned or non-patterned materials within themselves. Among the studies examined, there was no study that performed classification by bringing together the patterned and un-patterned fabrics in the Tilda dataset. For example, while the samples in the c1r1 folder in cd1 are classified within themselves, the c1r3, c2r2, c2r3 folders are also classified within themselves. In no study were the samples in the c1r1, c1r3, c2r2, c2r3 folders brought together and the methods were not tested as such. When a product that provides fabric control is produced, not only one fabric type, but also fabrics with a large number of patterns and textures, will pass through the control band. Therefore, testing on fabrics of different patterns and textures may be beneficial before converting the developed methods into products. The general contribution of this study is to create different sets by bringing together the patterned and un-patterned fabric samples in the Tilda dataset and to perform many tests on these sets.

Material and methods

Tilda dataset

The study is tested on the public part of the Tilda dataset. The Tilda dataset has images with 768 × 512 pixels. Images have been resized before processing operations. The dataset consists of two folders, such as cd1 and cd2 (Figure 1). Both folders consist of four groups (Table 1). While four groups in cd1 (c1r1, c1r3, c2r2, c2r3) consist of un-patterned fabric samples (Figure 2), the other four groups in cd2 (c3r1, c3r3, c4r1, c4r3) contain patterned samples (Figure 3). Each group is divided into eight subdirectories (e0, e1, e2, e3, e4, e5, e6, e7), and each contains 50 samples. Defect-free samples are found in e0, while samples with various types of defects are found in other subdirectories. The un-defected fabric is the fabric in which the texture repetition is not broken.

Figure 1.
Tilda dataset.

Table 1.
About the Tilda dataset

Un-patterned fabrics Patterned fabrics

Group Number of images Group Number of images

c1r1 400 (50 un-defected, 350 defected) c3r1 400 (50 un-defected, 350 defected)

Classes Classes

e0, e1, e2, e3, e4, e5, e6, e7 e0, e1, e2, e3, e4, e5, e6, e7

c1r3 400 (50 un-defected, 350 defected) c3r3 400 (50 un-defected, 350 defected)

Classes Classes

e0, e1, e2, e3, e4, e5, e6, e7 e0, e1, e2, e3, e4, e5, e6, e7

c2r2 400 (50 un-defected, 350 defected) c4r1 400 (50 un-defected, 350 defected)

Classes Classes

e0, e1, e2, e3, e4, e5, e6, e7 e0, e1, e2, e3, e4, e5, e6, e7

c2r3 400 (50 un-defected, 350 defected) c4r3 400 (50 un-defected, 350 defected)

Classes Classes

e0, e1, e2, e3, e4, e5, e6, e7 e0, e1, e2, e3, e4, e5, e6, e7

SUM 1600 (200 un-defected, 1400 defected) SUM 1600 (200 un-defected, 1400 defected)

Figure 2.
Un-patterned fabric samples of the Tilda dataset: (a) c1r1; (b) c1r3; (c) c2r2 and (d) c2r3.

Figure 3.
Patterned fabric samples of the Tilda dataset: (e) c3r1; (f) c3r3; (g) c4r1 and (h) c4r3.

CNN-based feature extraction

Deep learning is a sub-group of machine learning and is a multi-layered approach.³¹ There are many studies using deep learning in the literature. Deep learning is used in various fields, such as image processing, video processing, signal processing, object recognition, the defense industry, and robotics. It uses many layers of nonlinear processing units for feature extraction and conversion. The output from each previous layer is used by each subsequent layer as input. There are many different types of deep learning architectures established by increasing the number of layers in ANNs. The CNN is one of these architectures. The CNN is a multi-layer ANN model created specifically for computer vision applications. It is an approach that provides higher performance compared to other classification methods. On the other hand, the drawback of the CNN is the requirement for powerful hardware resources. The basic layers of CNN-based models are listed as in Table 2. In our study, we tested the performances of four CNN-based models, namely ResNet18, ResNet50, GoogLeNet, and AlexNet (Figure 4).

Table 2.
Basic layers of the convolutional neural network (CNN)

Layer Description

Convolution A new image is created in the convolution layer by extracting more specific features in the image. Filters have smaller dimensions than input data.

Activation function The activation functions of convolutional neural networks are crucial. The most often used activation function is the ReLU. Negative values in the input data are set to 0 in this layer, allowing the network to learn faster.

Pooling Pooling is a process used to reduce sizes in deep learning models to avoid overfitting and reduce time complexity.

Fully connected layer A fully connected layer is a one-dimensional matrix that is connected to all of the neurons in the layer before it. This layer is utilized to optimize class scores and is usually found near the end of the CNN architecture.

ReLU: rectified linear unit.

Figure 4.
Convolutional neural network (CNN) models used for our study.³²

ResNet

ResNet is a pre-trained CNN algorithm.³³ It has been trained on more than one million images. ResNet is the abbreviated form of residual network. It has different versions, such as ResNet18, ResNet50, ResNet101, and ResNet152. They include 18, 50, 101, and 152 layers, respectively. As the number of layers increases, the accuracy rate increases, and the execution time also increases. The sizes of input images must be 224-by-224 for the ResNet architecture. ResNet18 and ResNet50 have been used in this study.

GoogLeNet

GoogleNet (also known as Inception v1) is a pre-trained CNN algorithm, like ResNet18.³⁴ The sizes of input images must be 224-by-224 for GoogLeNet. It has an architecture consisting of 22 deeper layers with fewer parameters compared to other networks.³⁵ Therefore, it gives higher successful results in less time.

AlexNet

The AlexNet architecture, which won the ImageNet 2012 competition, is a type of CNN designed by Krizhevsky et al.³⁶ AlexNet is very similar to the LeNet network. However, it differs from LeNet in that it has more layers. The input dimensions should be 227 × 227 × 3 for AlexNet. Local response normalization (LRN) and dropout are new additions to this network.³⁷

Maximum relevance and minimum redundancy-based feature selection

The maximum relevance and minimum redundancy (mRMR) method is developed by Peng et al.³⁸ Its main goal is to choose the best features. The mRMR feature set is generated by maximizing D(S, y) and minimizing R(S) for a feature set S. Equations (1) and (2) must be met concurrently:
$\max D (S, y) = \frac{1}{| S |} \sum_{x_{i} \in S} I (x_{i}, y)$
(1)
$\min R (S) = \frac{1}{{| S |}^{2}} \sum_{x_{i}, x_{j} \in S} I (x_{i}, x_{j})$
(2)

Machine learning-based classification

Ensemble learning

Ensemble learning (EL) is a method for performing classification based on predictions and decisions from multiple classifiers.³⁹ It uses more than one classifier's information at the same time to apply each classifier's conclusion by consensus. This method outperforms a single classifier in most cases. The fact that the classifiers' mistakes differ from one another improves ensemble classification performance. Different subsets of the training dataset are used to achieve differences in the classifiers' predictions. To produce and train subsets of the training dataset, the bootstrap approach is utilized.

The predictions from the trained networks must be combined to arrive at an outcome. The choice of the right combined approach for the predictions has an impact on the classification performance in the EL method. The selection of the proper approach for the classifiers should be considered when choosing the combination technique. According to the combined procedures, sample selection for the training dataset, and processing processes, there are many EL approaches. Bagging, boosting, and voting are examples of these approaches.

k-nearest neighbor

The KNN algorithm is one of the simplest and most widely used classification algorithms. A k value is determined for the operation of the algorithm. This k value means the number of elements to look at for classification. The distances between the sample element and the other elements in the dataset are calculated using different distance metrics.⁴⁰ Euclidean, Manhattan, and Minkowski are among the most known distance metrics. In Table 3, their equations are given for the m feature. In these equations, s is the sample and n is the neighbor. In the equation of the Minkowski metric, q is an integer number greater than 0 and the value for q is determined by experiments. To estimate which class the sample element in the dataset belongs to, the nearest k-neighbors of the element are used. The element is assigned to the class to which most neighbors belong.

Table 3.
Distance metrics

Euclidean $\sqrt{\sum_{i = 1}^{m} {(s_{i} - n_{i})}^{2}}$

Manhattan $\sum_{i = 1}^{m} | s_{i} - n_{i} |$

Minkowski ${(\sum_{i = 1}^{m} {(s_{i} - n_{i})}^{q})}^{q}$

Support vector machine

The SVM is a machine learning technique proposed for classification problems in datasets where the patterns between variables are not known. It is a non-parametric classifier. This technique is also used for regression analysis, such as decision trees (DTs).

The dataset is separated into two classes: the training set and the test set. In this technique, the optimal hyperplane separating the classes is found using a labeled training set.⁴¹ There may be more than one plane separating the two classes. An optimal hyperplane is the farthest plane to the nearest data points of the classes.

If the problem is two-dimensional, the hyperplane is a line. As the number of dimensions increases, it becomes difficult to find the optimal hyperplane.

Proposed system: multi-feature fusion

The ultimate objective of a classification system is to create the best classification model for a given task, such as classification of fabrics. There is more than one metric for a system to be considered successful. In addition to statistical-based performance metrics, such as accuracy, precision, recall, and the f1-measure, the system's classification time performance of a sample is also important.

An essential stage in an image classification system is feature extraction. The performance of the entire system is impacted by the choice of feature extractor. It is thought that statistical-based performance can be improved when the features of two different types of CNN-based models are combined. Simply concatenating the separately extracted features using multiple models has been named feature-level fusion. Thus, it is aimed to increase the system performance and to take advantage of the different features of different models.

The features extracted using Method 1 are represented by vector X, and the features extracted using Method 2 are represented by vector Y. The feature fusion is obtained by adding the Y vector after the last element of the X vector (Equation (3)). The visualized version of the proposed system is in Figure 5:
$F = (X, Y) = (x_{1}, x_{2}, \dots, x_{n}, y_{1}, y_{2}, \dots, y_{m})$
(3)

Figure 5.
The procedure of the method proposed.

Multi-feature fusion increases the dimensionality of feature space, which leads to the problems of dimensionality and overfitting.⁴² As a result, combining all of the features in this level is not possible. Therefore, mRMR-based feature selection is used after feature fusion to reduce feature size and computational cost. Then, the selected features are classified using a classifier. Three algorithms, namely EL, KNN, and the SVM, have been tested in the classification step.

Experimental details

Evaluation metrics

In this study, the experimental results have been compared in terms of precision, recall, and F-measure. These indicators are calculated using the true positive (TP), true negative (TN), false positive (FP), and false negative (FN). Precision investigates how many fabrics labeled defected are defected. Recall investigates how many TPs are correctly identified. The harmonic mean of precision and recall gives the F-measure.

The models' classification performance has been evaluated using 10-fold cross-validation. Thus, the dataset is split into ten equal parts as D1, D2, …, D10.⁴³ The training and testing processes are repeated 10 times. Partition Di for iteration i is used for testing while the remaining partitions are used for training (Figure 6). The arithmetic average of the results obtained from each partition gives the success of the method.

Figure 6.
Ten-fold cross-validation.

Preliminaries to experiments

If CNN-based methods are used only for feature extraction, there is no need to split the dataset into training and testing. So, the entire database may be used for training in such studies. However, 70% of the database is reserved for training and 30% for testing to make comparisons in this study.

Classification of features drawn from previous layers of the CNN is generally less successful. Therefore, the features taken from the last layers were used in this study. Layers of ‘pool5,’ ‘avg_pool,’ ‘pool5-drop_7x7_s1,’ and ‘pool5’ have been used to extract the features for the methods of ResNet18, ResNet50, GoogLeNet, and AlexNet, respectively.

Results and evaluations

In this section, the Tilda dataset has been handled in three different ways, as shown in Table 4 (the sets of un-patterned fabrics, patterned fabrics, and mixed). Each of the three sets consists of two different classes, one containing defected fabrics and one containing un-defected fabrics. The sets of c1r1, c1r3, c2r2, and c2r3 in the cd1 folder have been brought together to create an un-patterned fabric dataset. There are 1400 defected images and 200 un-defected images. Among the un-patterned fabric images, it is attempted to classify fabrics as defected/un-defected (Figure 7). The sets of c3r1, c3r3, c4r1, and c4r3 in the cd2 folder have been brought together to create a patterned fabric dataset. There are 1400 defected images and 200 un-defected images. Among the patterned fabric images, fabrics, it is attempted to classify fabrics as defected/un-defected. The third set consists of all the fabric images in both the cd1 and cd2 folders. There are 2800 defected images and 400 un-defected images. Among the fabric images (un-patterned and patterned), it is attempted to classify fabrics as defected/un-defected. The main goal of this study is to see how well un-patterned, patterned, and both un-patterned and patterned fabrics in the sets can be classified. The success of the system in distinguishing defected fabrics from un-defected fabrics has been investigated in a set with different fabric types. This makes this study closer to a real-life problem compared to other studies.

Table 4.
Details

Dataset Number of elements

Un-patterned fabrics of the Tilda dataset 1600 (200 un-defected, 1400 defected)

Patterned fabrics of the Tilda dataset 1600 (200 un-defected, 1400 defected)

Mix of the Tilda dataset 3200 (400 un-defected, 2800 defected)

Figure 7.
Sample visualization for defect detection. CNN: convolutional neural network.

The experiments in this study have been carried out considering two different cases. In Case 1, features have been extracted with the help of a CNN-based model and classified after feature selection using mRMR. In Case 2, the performance of the proposed feature fusion process in the study has been investigated.

Case 1: using the CNN and machine learning (ML)

Table 5 shows the performances of the methods. When a classifier-based comparison is performed, the average f-measures for EL, KNN, and the SVM are 0.9374, 0.9281, and 0.9357, respectively. It can be said that the patterned fabrics dataset has the lowest classification success rate compared to the others. While the average f-measure value of patterned fabrics is 0.9297, the average f-measure value of non-patterned fabrics is 0.9363, and the average f-measure value of mixed fabrics is 0.9378.

Table 5.
Classification results (%)

Method Metric Un-patterned fabrics Patterned fabrics Mix Method Metric Un-patterned fabrics Patterned fabrics Mix

ResNet18 & EL Precision 88.58 87.53 87.54 GoogLeNet & EL Precision 98.57 99.57 99.79

Recall 99.21 99.29 99.96 Recall 88.62 87.89 87.47

F-measure 93.59 93.04 99.33 F-measure 93.34 93.37 93.22

ResNet18 & KNN Precision 89.11 87.79 88.76 GoogLeNet & KNN Precision 99.57 87.79 97.11

Recall 98.28 99.64 98.21 Recall 87.45 91.10 89.29

F-measure 93.47 93.34 93.25 F-measure 93.12 89.41 93.04

ResNet18 & SVM Precision 88.45 87.66 88.01 GoogLeNet & SVM Precision 99.14 99.93 99.68

Recall 99.64 99.93 99.96 Recall 89.95 87.48 88.46

F-measure 93.71 93.39 93.61 F-measure 94.32 93.29 93.74

ResNet50 & EL Precision 88.14 87.67 99.64 AlexNet & EL Precision 87.40 87.58 87.50

Recall 98.28 99.50 87.98 Recall 99.14 99.71 99.79

F-measure 92.93 93.21 93.45 F-measure 92.90 93.25 9324

ResNet50 & KNN Precision 91.15 88.13 97.43 AlexNet & KNN Precision 87.93 87.53 87.48

Recall 94.92 99.21 88.68 Recall 98.43 99.79 98.61

F-measure 93.00 93.34 92.85 F-measure 92.88 93.26 92.71

ResNet50 & SVM Precision 88.88 87.77 99.64 AlexNet & SVM Precision 87.48 87.48 87.46

Recall 99.43 99.93 88.23 Recall 99.93 99.93 99.96

F-measure 93.86 93.46 93.59 F-measure 93.23 93.29 93.29

EL: ensemble learning; KNN: k-nearest neighbors; SVM: support vector machine.

Since classification was performed for 100 features as a result of feature selection in this study, classification time comparisons only show the performance of the classifiers. Average classification times for EL, KNN, and the SVM are 12, 5.43, and 22.52 seconds, respectively (Table 6). For only 100 features, the SVM takes an average of 22.52 seconds to classify, which may be problematic for larger datasets. It is an interesting detail here that the classification times of the classifiers for the mixed dataset are higher (18.72 seconds) compared to the un-patterned (10.09 seconds) and patterned datasets (11.13 seconds).

Table 6.
Classification times (seconds)

Method Un-patterned fabrics Patterned fabrics Mix Method Un-patterned fabrics Patterned fabrics Mix

ResNet18 & EL 8.91 10.58 18.66 GoogLeNet & EL 11.53 9.14 16.6

ResNet18 & KNN 4.38 5.36 6.99 GoogLeNet & KNN 4.94 5.98 6.59

ResNet18 & SVM 8.82 10.79 26.15 GoogLeNet & SVM 13.85 16.11 27.2

ResNet50 & EL 10.39 11.4 15.83 AlexNet & EL 8.39 10.72 11.79

ResNet50 & KNN 4.99 4.45 6.22 AlexNet & KNN 4.94 4.64 5.68

ResNet50 & SVM 13.31 14.85 28.79 AlexNet & SVM 26.59 29.59 54.15

EL: ensemble learning; KNN: k-nearest neighbors; SVM: support vector machine.

Case 2: using feature fusion and machine learning (ML)

When the performance results given in Table 7 are examined, it is seen that classification can be performed with an average f-measure rate of about 93.34%. The average f-measure rates of the classifiers of EL, KNN, and the SVM are 93.20%, 93.10%, and 93.72%, respectively. The average f-measure rate of each fusion combination is between 93% and 94%. The average f-measure rate of ResNet18 & GoogLeNet-based classification is 94.37% for the mixed dataset.

Table 7.
Classification results for feature fusion (%)

EL-based classification KNN-based classification SVM-based classification

Method Metric Un-patterned fabrics Patterned fabrics Mix Un-patterned fabrics Patterned fabrics Mix Un-patterned fabrics Patterned fabrics Mix

ResNet18 & ResNet50 Precision 87.54 87.77 87.63 89.99 8794 89.04 88.46 87.88 88.29

Recall 98.93 99.43 99.93 96.35 99.00 97.82 99.21 99.93 99.93

F-measure 92.89 93.24 93.38 93.06 93.14 93.22 93.53 93.52 93.75

ResNet18 & GoogLeNet Precision 87.50 87.59 87.51 90.43 87.59 89.15 89.85 87.77 93.40

Recall 99.07 99.86 99.86 95.93 99.36 98.07 99.36 99.93 99.68

F-measure 92.93 93.32 93.28 93.10 93.10 93.4 94.37 93.18 96.44

ResNet18 & AlexNet Precision 87.86 87.83 87.52 89.53 87.90 88.6 88.20 87.48 87.57

Recall 98.78 99.5 99.93 97.21 99.64 97.78 99.36 99.93 99.96

F-measure 93.00 93.3 93.31 93.21 93.40 92.96 93.45 93.29 93.36

ResNet50 & GoogLeNet Precision 87.87 88.58 88.04 89.81 87.96 88.93 87.82 87.70 88.83

Recall 9843 98.71 99.75 95.71 98.71 98.5 99.50 99.86 99.75

F-measure 92.85 93.37 93.53 92.67 93.03 93.47 93.30 93.39 93.97

ResNet50 & AlexNet Precision 88.52 87.66 87.50 88.06 87.63 88.87 89.57 88.31 88.18

Recall 98.64 99.43 99.86 95.93 99.29 98.71 99.43 99.86 99.68

F-measure 93.31 93.17 93.27 91.83 93.10 93.53 94.24 93.73 93.58

GoogLeNet & AlexNet Precision 87.49 87.65 87.43 87.47 87.52 87.63 87.48 87.55 87.49

Recall 99.50 99.43 99.68 99.79 99.71 99.25 99.86 99.93 99.96

F-measure 93.11 93.17 93.15 93.22 93.22 93.08 93.26 93.33 93.31

EL: ensemble learning; KNN: k-nearest neighbors; SVM: support vector machine.

Average classification times for EL, KNN, and the SVM are 11.47, 5.45, and 19.86 seconds, respectively (Table 8). The classification times of the classifiers for the mixed dataset are higher (16.54 seconds) compared to the un-patterned (9.71 seconds) and patterned datasets (9.86 seconds).

Table 8.
Classification times for feature fusion (seconds)

EL-based classification KNN-based classification SVM-based classification

Un-patterned fabrics Patterned fabrics Mix Un-patterned fabrics Patterned fabrics Mix Un-patterned fabrics Patterned fabrics Mix

ResNet18 & ResNet50 9.37 11.44 15.1 4.43 5.03 6.89 13.75 15.5 27.21

ResNet18 & GoogLeNet 7.68 10.84 15.23 2.54 4.43 5.14 13.32 15.81 26.46

ResNet18 & AlexNet 12.72 11.88 17.13 4.76 4.12 5.77 17.17 16.22 30.18

ResNet50 & GoogLeNet 9.43 9.9 15.48 4.95 5.06 5.77 15.35 15.89 27.08

ResNet50 & AlexNet 9.38 10.8 13.23 4.44 4.18 5.06 13.81 14.37 24.92

GoogLeNet & AlexNet 7.85 7.6 11.42 4.07 3.62 5.75 19.73 10.78 39.88

EL: ensemble learning; KNN: k-nearest neighbors; SVM: support vector machine.

The results of Cases 1 and 2 are given together in Figure 8. In the first case, ResNet18 is the network with the highest average f-measure value (0.9408), while GoogLeNet has the lowest (0.9298). When the features extracted with ResNet18 are classified with EL, 95.32% success is achieved (the highest success). The lowest success was obtained with the combination of GoogLeNet & KNN (92.98%). In the second case, a success of 94.66% has been achieved when the features obtained from the fusion of ResNet18 & GoogleNet are classified with the SVM. In both cases, it is seen that generally the lowest performance is obtained when KNN is used (on average 92.98%). The SVM is the classifier with the highest average in both cases (93.66%), while EL is second in performance (93.42%). As can be seen from this graph, the results obtained on average are quite successful.

Figure 8.
Comparison based on the f-measure. EL: ensemble learning; KNN: k-nearest neighbors; SVM: support vector machine.

Conclusions and future works

Given the recent developing technology, automatic fabric defect detection systems have attracted a great deal of attention. They are required for a variety of reasons, including increased product quality.

Particularly in recent years, the use of CNN-based models has been very popular not only in this field but also in all fields. It is obvious that the CNN has advantages as well as disadvantages. However, the disadvantages of the CNN are not enough to stain its popularity. In the literature, it is seen that CNN models developed by considering their weak points are presented. This study aims to present a system that will give results of feature extraction and classification in less time and is at least as successful as the CNN. For these purposes, a comprehensive analysis has been carried out in this study.

The Tilda database has been used for testing the methods. Subdirectories other than e0 have been collected in one directory, since it has been attempted to classify fabric samples as defected or non-defected. As a consequence, for un-patterned fabrics, we have 200 non-defected images and 1400 defected images. On the other hand, the number of images we have for patterned fabrics is the same as the number of images for un-patterned fabrics. In addition to all of these, the study looked into the performance of classification in a dataset that included both patterned and un-patterned fabrics. This dataset is called ‘mix’ and it consists of 400 un-defected images and 2800 defected images from the preceding two datasets. Images with and without patterns, as well as various textures, have all been identified as defected or un-defected in this way. The performances of three sets (un-patterned, patterned, mixed) have been analyzed and compared separately. The study consists of two cases. One feature extraction method and one classifier are used in Case 1, while fusion features are obtained and classified in Case 2. In both cases, the features obtained in the feature extraction process have been passed through the feature selection process before classification.

Different approaches using four CNN-based models, namely ResNet18, ResNet50, GoogLeNet, and AlexNet, have been used in the feature extraction step, while EL, KNN, and the SVM have been used in the classification step. In this study, unlike other studies, feature fusion has been used for feature extraction. In feature fusion for Case 2, binary combinations of ResNet18, ResNet50, GoogLeNet, and AlexNet (ResNet18 & ResNet50, ResNet18 & GoogLeNet, ResNet18 & AlexNet, ResNet50 & GoogLeNet, ResNet50 & AlexNet, GoogLeNet & AlexNet) are preferred.

If we reach the following conclusions for the cases mentioned above.
ResNet18 & EL has the highest performance in the first case (95.32%). Considering their average performances, EL (93.74%) and the SVM (93.57%) have close values while KNN's value is slightly lower (92.81%). KNN is a classifier that is sensitive to variables, and therefore not robust.

In the second case, ResNet18 & GoogLeNet & SVM achieved 94.66%, on average. The SVM is the highest performing classifier (93.72%), and EL is the second (93.20%). Here, as in Case 1, the lowest performance was obtained when using KNN (93.10%).

The following issues can be solved in the future.
Converting the system into a product as a real-time defect detecting system on weaving machines will close the gap in this area and will be economical, as it is a domestic solution.

It is obvious that there is a dataset problem in this area. In the future, it is aimed to create a large database containing a large number of fabric defect samples.

Un-patterned fabrics	Patterned fabrics
c1r1	400 (50 un-defected, 350 defected)	c3r1	400 (50 un-defected, 350 defected)
Classes	Classes
e0, e1, e2, e3, e4, e5, e6, e7	e0, e1, e2, e3, e4, e5, e6, e7
c1r3	400 (50 un-defected, 350 defected)	c3r3	400 (50 un-defected, 350 defected)
Classes	Classes
e0, e1, e2, e3, e4, e5, e6, e7	e0, e1, e2, e3, e4, e5, e6, e7
c2r2	400 (50 un-defected, 350 defected)	c4r1	400 (50 un-defected, 350 defected)
Classes	Classes
e0, e1, e2, e3, e4, e5, e6, e7	e0, e1, e2, e3, e4, e5, e6, e7
c2r3	400 (50 un-defected, 350 defected)	c4r3	400 (50 un-defected, 350 defected)
Classes	Classes
e0, e1, e2, e3, e4, e5, e6, e7	e0, e1, e2, e3, e4, e5, e6, e7
SUM	1600 (200 un-defected, 1400 defected)	SUM	1600 (200 un-defected, 1400 defected)

Layer	Description
Convolution	A new image is created in the convolution layer by extracting more specific features in the image. Filters have smaller dimensions than input data.
Activation function	The activation functions of convolutional neural networks are crucial. The most often used activation function is the ReLU. Negative values in the input data are set to 0 in this layer, allowing the network to learn faster.
Pooling	Pooling is a process used to reduce sizes in deep learning models to avoid overfitting and reduce time complexity.
Fully connected layer	A fully connected layer is a one-dimensional matrix that is connected to all of the neurons in the layer before it. This layer is utilized to optimize class scores and is usually found near the end of the CNN architecture.

Dataset	Number of elements
Un-patterned fabrics of the Tilda dataset	1600 (200 un-defected, 1400 defected)
Patterned fabrics of the Tilda dataset	1600 (200 un-defected, 1400 defected)
Mix of the Tilda dataset	3200 (400 un-defected, 2800 defected)

Method	Metric	Un-patterned fabrics	Patterned fabrics	Mix	Method	Metric	Un-patterned fabrics	Patterned fabrics	Mix
ResNet18 & EL	Precision	88.58	87.53	87.54	GoogLeNet & EL	Precision	98.57	99.57	99.79
Recall	99.21	99.29	99.96	Recall	88.62	87.89	87.47
F-measure	93.59	93.04	99.33	F-measure	93.34	93.37	93.22
ResNet18 & KNN	Precision	89.11	87.79	88.76	GoogLeNet & KNN	Precision	99.57	87.79	97.11
Recall	98.28	99.64	98.21	Recall	87.45	91.10	89.29
F-measure	93.47	93.34	93.25	F-measure	93.12	89.41	93.04
ResNet18 & SVM	Precision	88.45	87.66	88.01	GoogLeNet & SVM	Precision	99.14	99.93	99.68
Recall	99.64	99.93	99.96	Recall	89.95	87.48	88.46
F-measure	93.71	93.39	93.61	F-measure	94.32	93.29	93.74
ResNet50 & EL	Precision	88.14	87.67	99.64	AlexNet & EL	Precision	87.40	87.58	87.50
Recall	98.28	99.50	87.98	Recall	99.14	99.71	99.79
F-measure	92.93	93.21	93.45	F-measure	92.90	93.25	9324
ResNet50 & KNN	Precision	91.15	88.13	97.43	AlexNet & KNN	Precision	87.93	87.53	87.48
Recall	94.92	99.21	88.68	Recall	98.43	99.79	98.61
F-measure	93.00	93.34	92.85	F-measure	92.88	93.26	92.71
ResNet50 & SVM	Precision	88.88	87.77	99.64	AlexNet & SVM	Precision	87.48	87.48	87.46
Recall	99.43	99.93	88.23	Recall	99.93	99.93	99.96
F-measure	93.86	93.46	93.59	F-measure	93.23	93.29	93.29

Method	Un-patterned fabrics	Patterned fabrics	Mix	Method	Un-patterned fabrics	Patterned fabrics	Mix
ResNet18 & EL	8.91	10.58	18.66	GoogLeNet & EL	11.53	9.14	16.6
ResNet18 & KNN	4.38	5.36	6.99	GoogLeNet & KNN	4.94	5.98	6.59
ResNet18 & SVM	8.82	10.79	26.15	GoogLeNet & SVM	13.85	16.11	27.2
ResNet50 & EL	10.39	11.4	15.83	AlexNet & EL	8.39	10.72	11.79
ResNet50 & KNN	4.99	4.45	6.22	AlexNet & KNN	4.94	4.64	5.68
ResNet50 & SVM	13.31	14.85	28.79	AlexNet & SVM	26.59	29.59	54.15

		EL-based classification	KNN-based classification	SVM-based classification
ResNet18 & ResNet50	Precision	87.54	87.77	87.63	89.99	8794	89.04	88.46	87.88	88.29
Recall	98.93	99.43	99.93	96.35	99.00	97.82	99.21	99.93	99.93
F-measure	92.89	93.24	93.38	93.06	93.14	93.22	93.53	93.52	93.75
ResNet18 & GoogLeNet	Precision	87.50	87.59	87.51	90.43	87.59	89.15	89.85	87.77	93.40
Recall	99.07	99.86	99.86	95.93	99.36	98.07	99.36	99.93	99.68
F-measure	92.93	93.32	93.28	93.10	93.10	93.4	94.37	93.18	96.44
ResNet18 & AlexNet	Precision	87.86	87.83	87.52	89.53	87.90	88.6	88.20	87.48	87.57
Recall	98.78	99.5	99.93	97.21	99.64	97.78	99.36	99.93	99.96
F-measure	93.00	93.3	93.31	93.21	93.40	92.96	93.45	93.29	93.36
ResNet50 & GoogLeNet	Precision	87.87	88.58	88.04	89.81	87.96	88.93	87.82	87.70	88.83
Recall	9843	98.71	99.75	95.71	98.71	98.5	99.50	99.86	99.75
F-measure	92.85	93.37	93.53	92.67	93.03	93.47	93.30	93.39	93.97
ResNet50 & AlexNet	Precision	88.52	87.66	87.50	88.06	87.63	88.87	89.57	88.31	88.18
Recall	98.64	99.43	99.86	95.93	99.29	98.71	99.43	99.86	99.68
F-measure	93.31	93.17	93.27	91.83	93.10	93.53	94.24	93.73	93.58
GoogLeNet & AlexNet	Precision	87.49	87.65	87.43	87.47	87.52	87.63	87.48	87.55	87.49
Recall	99.50	99.43	99.68	99.79	99.71	99.25	99.86	99.93	99.96
F-measure	93.11	93.17	93.15	93.22	93.22	93.08	93.26	93.33	93.31

	EL-based classification	KNN-based classification	SVM-based classification
ResNet18 & ResNet50	9.37	11.44	15.1	4.43	5.03	6.89	13.75	15.5	27.21
ResNet18 & GoogLeNet	7.68	10.84	15.23	2.54	4.43	5.14	13.32	15.81	26.46
ResNet18 & AlexNet	12.72	11.88	17.13	4.76	4.12	5.77	17.17	16.22	30.18
ResNet50 & GoogLeNet	9.43	9.9	15.48	4.95	5.06	5.77	15.35	15.89	27.08
ResNet50 & AlexNet	9.38	10.8	13.23	4.44	4.18	5.06	13.81	14.37	24.92
GoogLeNet & AlexNet	7.85	7.6	11.42	4.07	3.62	5.75	19.73	10.78	39.88

Footnotes

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iDs

Fatma Günseli Yaşar Çiklaçandir

Semih Utku

Hakan Özdemir

References

Srinivasan

Dastoor

Radhakrishnaiah

, et al. FDAS: a knowledge-based framework for analysis of defects in woven textile structures. J Text Inst 1992; 83: 431–448.

Rasheed

Zafar

Rasheed

, et al. Fabric defect detection using computer vision techniques: a comprehensive review. Math Probl Eng 2020; 2020: 1–24.

Kısaoğlu

Ö.

Orta büyüklükte bir dokuma işletmesinde istatistiksel kalite kontrol sisteminin kurulması. MSc Thesis, Dokuz Eylül Üniversity, 2002.

Kuo

CFJ

Lee

CJ.

A back propagation neural network for recognizing fabric defects. Text Res J 2003; 73: 147–151.

Huang

Chen

IC.

Neural fuzzy classification for fabric defects. Text Res J 2001; 71: 220–224.

Kumar

Neural network based detection of local textile defects. Pattern Recognit 2003; 36: 1645–1659.

Behera

Mani

MP.

Characterization and classification of fabric defects using discrete cosine transformation and artificial neural network. Ind J Fibre Text Res 2007; 32: 421–426.

Büyükkabasakal

Detection and classification of fabric defects. İzmir: Ege University.

Çelik

Hİ

Dülger

Topalbekiroğlu

Fabric defect detection using linear filtering and morphological operations. Ind J Fibre Text Res 2014; 39: 254–259.

10.

Jmali

Zitouni

Sakli

Fabrics defects detecting using image processing and neural networks. Inform Commun Technol Innovat Appl. Epub ahead of print 6–8 March 2014. DOI: 10.1109/ICTIA.2014.7883765.

11.

Rebhi

Benmhammed

Abid

, et al. Fabric defect detection using local homogeneity analysis and neural network. J Photon 2015; 2015: 1–15.

12.

Wei

Hao

Tang

, et al. A new method using the convolutional neural network with compressive sensing for fabric defect classification based on small sample sizes. Text Res J 2019; 89: 3539–3555.

13.

Şeker

Peker

Yüksek

, et al. Fabric defect detection using deep learning. In: 24th signal processing and communication application conference (SIU), Zonguldak, Turkey, 16–19 May 2016, pp. 1437–1440. IEEE.

14.

Basturk

Ketencioglu

Yugnak

, et al. Inspection of defects in fabrics using Gabor wavelets and principle component analysis. In: 2007 9th international symposium on signal processing and its applications, Sharjah, United Arab Emirates, 12–15 February 2007, pp. 1–4. IEEE.

15.

Bissi

Baruffa

Placidi

, et al. Automated defect detection in uniform and structured fabrics using Gabor filters and PCA. J Vis Commun Image Rep 2013; 24: 838–845.

16.

Kure

Biradar

Bhangale

KB.

Local neighborhood analysis for fabric defect detection. In: international conference on information, communication, instrumentation and control (ICICIC), Indore, India, 17–19 August 2017, pp. 1–5. IEEE.

17.

Sezer

Erçil

Ertuzun

Using perceptual relation of regularity and anisotropy in the texture with independent component model for defect detection. Pattern Recognit 2007; 40: 121–133.

18.

Kaynar

Işik

Görmez

, et al. Fabric defect detection with LBP-GLMC. In: international artificial intelligence and data processing symposium (IDAP), Malatya, Turkey, 16–17 September 2017, pp. 1–5. IEEE.

19.

Ben Salem

Nasri

Woven fabric defects detection based on texture classification algorithm. In: 8th international multi-conference on systems, signals & devices, Tunisia, 22–25 March 2011, 43(24). IAEA.

20.

Murino

Bicego

Rossi

IA.

Statistical classification of raw textile defects. In: proceedings of the 17th international conference on pattern recognition, 2004.

21.

Chandra

Majumdar

Sarkar

Feature extraction and classification of woven fabric using optimized Haralick parameters: a rough set based approach. In: 2016 2nd international conference on control, instrumentation, energy & communication (CIEC), 28–30 January 2016, pp. 541–545. IEEE.

22.

Makaremi

Razmjooy

Ramezani

A new method for detecting texture defects based on modified local binary pattern. Sign Image Video Proc 2018; 12: 1395–1401.

23.

Cuifang

Jiacheng

Fabric defect detection algorithm based on PHOG and SVM. Ind J Fibre Text Res 2020; 45: 123–126.

24.

Basibuyuk

Coban

Ertuzun

Model based defect detection problem: particle filter approach. In: 2008 3rd international symposium on communications, control and signal processing, March 2008, pp. 348–351. IEEE.

25.

Jing

Yang

Defect detection on patterned fabrics using distance matching function and regular band. J Eng Fibers Fabr 2015; 10: 155892501501000210.

26.

Zou

Zhang

, et al. Defect detection on the fabric with complex texture via dual-scale over-complete dictionary. J Text Inst 2016; 107: 743–756.

27.

Cao

Zhang

Wen

, et al. Fabric defect inspection using prior knowledge guided least squares regression. Multimed Tools Appl 2010; 76: 4141–4157.

28.

Liu

Zhang

, et al. Unsupervised segmentation and elm for fabric defect image classification. Multimed Tools Appl 2019; 78: 12421–12449.

29.

Jing

Zhang

HH.

Automatic fabric defect detection using a deep convolutional neural network. Color Technol 2019; 135: 213–223.

30.

Jeyaraj

Nadar

ERS.

Computer vision for automatic detection and classification of fabric defect employing deep learning algorithm. Int J Clothing Sci Technol 2019; 31: 510–521.

31.

Türkoğlu

Hanbay

Sivrikaya

, et al. Classification of apricot diseases by using deep convolution neural network. Bitlis Eren Univ J Sci 2021; 9: 334–345.

32.

Yaşar Çıklaçandır

Utku

Özdemir

Determination of various fabric defects using different machine learning techniques. J Text Inst 2023; 1–11.

33.

Zhang

Ren

, et al. Deep residual learning for image recognition. In: proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770–778.

34.

Szegedy

Liu

Jia

, et al. Going deeper with convolutions. In: proceedings of the IEEE conference on computer vision and pattern recognition, 2015, pp.1–9.

35.

Tellawi

Classification of malaria infected cells using inception V1. PhD Thesis, Near East University, 2019.

36.

Krizhevsky

Sutskever

Hinton

GE.

Imagenet classification with deep convolutional neural networks. Adv Neur Inform Proc Syst 2012; 25: 1–9.

37.

Alom

Taha

Yakopcic

, et al. The history began from AlexNet: a comprehensive survey on deep learning approaches. arXiv preprint arXiv:1803.01164 2018.

38.

Peng

Long

Ding

Feature selection based on mutual information criteria of max-dependency, max-relevance, and min-redundancy. IEEE Trans Pattern Anal Mach Intell 2005; 27: 1226–1238.

39.

Baran

Classification of multi-label data with machine learning methods. MSc Thesis, Sivas Cumhuriyet University, 2020.

40.

Chomboon

Chujai

Teerarassamee

, et al. An empirical study of distance metrics for k-nearest neighbor algorithm. In: proceedings of the 3rd international conference on industrial application engineering, Kitakyushu, Japan, 28–31 March 2015, pp. 280–285.

41.

Cortes

Vapnik

Support vector networks. Mach Learn 1995; 20: 1–25.

42.

Taheri

Fusion of hand-crafted descriptors with CNN-based features for facial age estimation. PhD Thesis, Turkey: Eastern Mediterranean University, 2019.

43.

Han

Pei

Kamber

, et al. Data mining: concepts and techniques. 3th ed. Massachusetts: Elsevier, 2011.

Un-patterned fabrics		Patterned fabrics
Group	Number of images	Group	Number of images
c1r1	400 (50 un-defected, 350 defected)	c3r1	400 (50 un-defected, 350 defected)
	Classes		Classes
	e0, e1, e2, e3, e4, e5, e6, e7		e0, e1, e2, e3, e4, e5, e6, e7
c1r3	400 (50 un-defected, 350 defected)	c3r3	400 (50 un-defected, 350 defected)
	Classes		Classes
	e0, e1, e2, e3, e4, e5, e6, e7		e0, e1, e2, e3, e4, e5, e6, e7
c2r2	400 (50 un-defected, 350 defected)	c4r1	400 (50 un-defected, 350 defected)
	Classes		Classes
	e0, e1, e2, e3, e4, e5, e6, e7		e0, e1, e2, e3, e4, e5, e6, e7
c2r3	400 (50 un-defected, 350 defected)	c4r3	400 (50 un-defected, 350 defected)
	Classes		Classes
	e0, e1, e2, e3, e4, e5, e6, e7		e0, e1, e2, e3, e4, e5, e6, e7
SUM	1600 (200 un-defected, 1400 defected)	SUM	1600 (200 un-defected, 1400 defected)

Euclidean	$\sqrt{\sum_{i = 1}^{m} {(s_{i} - n_{i})}^{2}}$
Manhattan	$\sum_{i = 1}^{m} \| s_{i} - n_{i} \|$
Minkowski	${(\sum_{i = 1}^{m} {(s_{i} - n_{i})}^{q})}^{q}$

The effects of fusion-based feature extraction for fabric defect classification

Abstract

Keywords

Related works

Material and methods

Tilda dataset

CNN-based feature extraction

ResNet

GoogLeNet

AlexNet

Maximum relevance and minimum redundancy-based feature selection

Machine learning-based classification

Ensemble learning

k-nearest neighbor

Support vector machine

Proposed system: multi-feature fusion

Experimental details

Evaluation metrics

Preliminaries to experiments

Results and evaluations

Case 1: using the CNN and machine learning (ML)

Case 2: using feature fusion and machine learning (ML)

Conclusions and future works

Footnotes

Declaration of conflicting interests

Funding

ORCID iDs

References