Micro-KTNet: Microstructure knowledge transfer learning for fiber masterbatch agglomeration recognition

Abstract

Fiber masterbatch production suffers from inherent agglomeration effect of high-concentration color masterbatches, negatively impacting color uniformity, particle dispersion, and thermal stability in fiber masterbatch, and ultimately the quality of fiber products. However, accurately recognizing and classifying the agglomeration in scanning electron microscopy (SEM) images remain a challenge due to the complexity of agglomeration status, difficulty in distinguishing micro-size agglomeration, and limited data availability. To address this challenge, this paper proposes a novel microstructure recognition architecture, named Micro-KTNet, which is designed for segmenting SEM images of fiber masterbatch aggregation. First, Micro-KTNet leverages transferable microstructural features to initialize the proposed network, mitigating the impact of limited data. Then, to handle multi-scale features in the microstructure, an encoding-decoding structure is constructed and skip connections are used to transfer the output features of the encoding layer. Finally, the decoder incorporates a novel attention module to effectively process both global and local features. To evaluate Micro-KTNet, a unique fiber masterbatch SEM database is built as the benchmark for micro-size agglomeration recognition. Experimental results indicate that Micro-KTNet surpasses existing state-of-the-art methods and improves the recognition precision from 75. 37% to 87. 04%.

Introduction

Fiber masterbatch, a fundamental component within the chemical fiber industry, is a blend of different materials. It mainly comprises high-level thermoplastic polymers, micro-nano functional powders, and various additives. The complex industrial fabrication process of fiber masterbatch encompasses multiple stages such as mixing raw material, extruding the mixture, cooling, and quality inspections. However, During the formation of fiber masterbatches, the agglomeration status of particles is unavoidable due to charge forces between particles and the action of dispersants. ¹ In the mixture formed by the raw materials, those particles, develop significant positive and negative charges on their surfaces. Then, particles of inconsistent sizes can accumulate and stick together due to the charge force between them. While inherent to the manufacturing process, particle agglomeration has a negative impact on the quality and performance of the final products. ² Traditionally, researchers have primarily evaluated agglomeration effects by focusing on filtration efficiency. ³ Additionally, industry experts in the fiber field can reply on their expertise to manually evaluate the agglomeration results through the analysis of the captured electron microscope images (SEM). ⁴ However, due to the low efficiency and high labor intensity associated with human inspection, automated detection based on machines or computers has drawn considerable attention in recent decades.

At present, deep learning employs multi-layer neural networks to extract high-level abstract features from data and achieves remarkable success. ⁵ It has significantly enhanced the performance of tasks such as image classification, ⁶ object detection, ⁷ and semantic segmentation, ⁸ owing to its self-learning and training capabilities. ⁹ In the field of fiber textiles, deep learning has shown remarkable efficacy in textile image classification, ¹⁰ optimizing chemical fiber processes, ¹¹ and online detection of fiber quality. ¹² For instance, Wei et al. ¹³ proposed an integrated learning framework called the bio-inspired visual integrated framework (BIVI-ML) for efficiently classifying multi-label textile defects. In this work, three bio-inspired visual mechanisms (the visual gain mechanism, the visual attention mechanism, and the visual memory mechanism) were proposed and built within the BIVI-ML. In addition, Tan et al. ¹⁴ proposed a scale and context information fusion network for multi type medical image segmentation, called SCIF-Net. SCIF-Net showcasing its superior capability in effectively capturing multi-scale features and spatial information, thereby enabling accurate segmentation of diverse human tissue structures. Additionally, Wu et al. ¹⁵ proposed a novel generative adversarial network (GAN)model, called SSGAN-ASP. SSGAN-ASP is designed to enhance fundus images by preserving anatomical structures and employing semi supervised learning, demonstrating improved visual quality and diagnostic accuracy. Moreover, Wei et al. ¹⁶ proposed a novel detection model to tackle the challenge of identifying small-scale textile defects, called Faster VG-RCNN. By incorporating attention-related visual gain mechanisms into Faster RCNN, this approach promises to enhance the accuracy and reliability of defect detection in textile manufacturing processes. Du et al. ¹⁷ applied an artificial intelligence technology into waste textile identification and sorting. Developing two online NIR qualitative identification models covering thirteen kinds of waste textiles using the convolutional neural network (CNN)and Baidu’s deep learning plat form PaddlePaddle. Furthermore, for the task of textile defect recognition with few labeled samples, a semi-supervised spatial-spectral neural network ¹⁸ was proposed, which could capture both spectral and spatial feature information from textile images. Experimental results showed that the textile defect recognition model achieved outstanding performance in a series of textile defect recognition challenges. Meanwhile, in terms of identifying the internal microstructure of fiber masterbatch, scanning electron microscopy (SEM) imaging is a well-established methodology at different length scales, allowing for an in-depth structural investigation and physical properties evaluation.

In Current research on SEM microstructure recognition, several deep neural network models have demonstrated impressive performance. For example, Quan et al. ¹⁹ pro posed a deep fully residual convolutional neural network for image segmentation in connect omics, called FusionNet. FusionNet introduces sum-based skip connections to deepen the network structure and enhance the segmentation accuracy across several electron microscopy segmentation datasets. Furthermore, Chen et al. ²⁰ proposed a pioneering Transformer-based Attention Guided Network to advance automatic medical image segmentation, called TransAttUnet. Integrating self-aware attention modules and multi scale skip connections, TransAttUnet adeptly captures long range contextual dependencies, thereby enhancing semantical segmentation accuracy. Additionally, Wei et al. ²¹ introduced a methodology utilizing deep learning feature pyramid networks (FPN)to tackle the complexities of multi-class fabric defect detection, addressing challenges such as intersecting defects and small-size anomalies. Meanwhile, Zhou et al. ²² developed a physics-informed deep learning framework to predict the strength of composite materials with microstructural uncertainties using RVE images generated by a random fiber packing algorithm. The framework significantly accelerates uncertainty analysis and outperforms traditional methods, with its effectiveness validated through systematic case studies and statistical cross-validation. Besides, Zeng et al. ²² proposed an improved neural network based on Unet for nuclei segmentation in histology images. Moreover, Zhang et al. ²³ proposed a very deep residual channel attention network (RCAN)for electron microscopy image super-resolution (SR). However, these models share a common limitation: data dependency. They rely on extensive annotated datasets for training. Acquiring high-quality annotations for SEM images can be challenging and time-consuming in practice. This limitation restricts the generalization capability and robustness of these models. To address this issue, transfer learning methods are often employed. For instance, Tamarina ²⁴ et al. proposed a transfer learning approach using Convolutional Neural Networks (CNNs) pre-trained on the ImageNet dataset. This method utilizes CNN layers as feature extractors to derive features from the ImageNet dataset, which are then passed to a support vector machine with a linear kernel. However, this method still faces some issues in practice. On one hand, there may be feature distribution differences and missing data labels between the source domain and the target domain. In such cases, it is necessary to identify and utilize the similarities between the source domain and the target domain to transfer knowledge from the source domain to the target domain. On the other hand, the knowledge learned in the source domain may negatively impact learning in the target domain.

In this paper, building upon the advancement of deep learning for SEM microstructure recognition, we explore the microstructure knowledge transfer learning-based agglomeration recognition network (Micro-KTNet), which has not been widely recognized and studied, but is of significant importance. The integration of several modules in the proposed model focuses on the dispersion superstructure and the action mechanism of parent particles. Furthermore, the proposed Micro-KTNet utilizes feature learning and complex modeling capabilities to segment aggregated particles, showcasing its recognition performance in fiber masterbatch industry process. The primary contributions of this paper are:

1. We propose a novel end-to-end framework for masterbatch agglomeration recognition with few labeled SEM images, which transforms the task of agglomeration recognition into a task based on microstructure knowledge transfer learning-based method.

2. We present a novel attention encoding-decoding structure, in which the encoder with skip connections is capable of producing multi-scale features and decoder with attention module can process both global and local features effectively.

3. We build a benchmark dataset for masterbatch agglomeration to evaluate the empirical performance of Micro-KTNet. Experimental results show that the proposed model achieves outstanding performance in masterbatch agglomeration recognition with few labeled SEM images.

The remainder of this paper is organized as follows. In Related works section, we briefly summarize the basic concept of transfer learning and encoder-decoder. Details of the proposed framework are presented in The proposed Micro-KTNe section. Performance comparisons between the proposed approach and current state-of-the-art approaches are described in Experimental evaluation and discussion section. The paper is concluded in Conclusions and future work section.”

Related works

Transfer learning

Transfer learning aims to improve the performance of a target task represented as $T_T$ in target domain $D_T$ , by leveraging knowledge from a source domain $D_S$ and a source task $T_S$ , where $D_S\ne D_T$ or $T_S\ne T_T$ . ²⁵ In this context, domain $D$ is closely intertwined with the data distribution for training. It is characterized by a feature space $\chi$ and a marginal probability distribution $P\left(X\right)$ , where $X=\left\{x_1,.\hspace{0.17em}.\hspace{0.17em}.,x_n\right\}\in \chi$ . Each unique domain requires its own distinctive feature space or marginal probability distribution. The task $T$ is learned from training data $T\left\{xi,yi\right\}$ , where $xi\in X$ and $yi\in Y$ . The objective of transfer learning, particularly in deep transfer learning, is to enhance the function $f_T(·)$ in the target task $T_T$ by incorporating knowledge from source tasks $T_S$ . Generally, these source tasks involve a larger source domain $D_S$ compared to the target domain $D_T$ . It is generally believed that the features extracted by the initial layers of neural networks are universal. Recently, transfer learning techniques were broadly classified into four types based on their applications: instance-based, mapping-based, network based, and adversarial-based. ²⁶ Among these, the most prevalent approach in practical applications is network-based transfer learning, capitalizing on the inherent structure of networks to enhance performance and adaptability. Research ²⁷ demonstrated that image representations learned by convolutional neural networks (CNN)using large-scale annotated datasets could effectively transfer to other visual recognition tasks with limited training data. As a result, many visual-related studies^28–30 commonly initialized parameters with models pre-trained on the large-scale ImageNet dataset and then fine-tuned or freeze specific parameters. However, in the task of ultrastructural image segmentation, Stuckner et al. ³¹ employed a pre-trained model on the large-scale MicroNet microscopy dataset as the initial parameters for the image segmentation network. For the transfer of heterogeneous neural networks, an encoder-decoder network is a commonly used architecture.

Encoder-decoder

The encoder-decoder is an important network structure in deep learning, which comprises two components: the encoder and the decoder. The encoder is responsible for transforming the input into an intermediate state known as the feature, while the decoder decodes the feature to produce the desired output. As the encoder-decoder is a generic framework rather than a specific model, it has the flexibility to handle various types of data.^32–34 Meanwhile, the design of diverse models can be facilitated by the encoder-decoder architecture. For example, recurrent neural networks (RNN) can be regarded as encoder-decoder structures. The encoder converts input text into vectors, and the decoder decodes the vectors into the desired outputs. Consequently, the RNN encoder-decoder architecture finds its application in sequence-to-sequence tasks, where the input and output are sequences of varying lengths. This approach is commonly known as the encoder-decoder Seq2Seq model. ³⁵ In this model, the RNN can be a simple RNN, a long short-term memory (LSTM), or a gated recurrent unit (GRU). For a simple RNN, each intermediate state, denoted by $H_t$ (encoder), is computed using the equation (1)

$H_t\left(encoder\right)=\phi \left(W_{HH}\times H_{t-1}+W_{HX}\times X_t\right)$

(1)where φ represents the activation function, $W_{HH}$ is the weight matrix for connecting intermediate states, and $W_{HX}$ is the weight matrix for connecting the input and intermediate states. The hidden state of the decoder, $H_t$ (decoder), is calculated as formula (2)

$H_t\left(\mathit{decoder}\right)=\phi \left(W_{HH}\times H_{t-1}\right)$

(2)

The decoder output is given by formula (3)

$Y_t=H_t\left(\mathit{decoder}\right)\times W_{HY}$

(3)where $W_{HY}$ is the weight matrix connecting the intermediate state and the decoder output. In addition to these examples, RNN-based encoder-decoder models have been utilized in various applications. For instance, Cho et al. ³² proposed an RNN encoder-decoder model that employs an RNN as the encoder, encoding a symbolic sequence into a fixed-length vector representation. Another RNN was employed as the decoder to decode the representation into another symbolic sequence, thus achieving machine translation. Additionally, Wang et al. ³² presented a generator that utilizes RNN to generate Chinese poetry. Furthermore, Serban et al. ³² introduced a hierarchical latent variable encoder-decoder model for generating dialogue, which involves a two-step generation process incorporating the sampling of latent variables and the generation of output sequences using a random latent variable. This model can capture long-term context and model sequences with a hierarchical structure. In a variation, Park et al. ³² replaced the recurrent neural network (RNN)with long short-term memory (LSTM)and proposed an encoder-decoder model based on LSTM. This model allows the sequence-to-sequence prediction of vehicle trajectories. Similarly, encoder-decoder architectures can also be applied to convolutional neural networks (CNN). In CNNs, images pass through convolutional layers, followed by linear layers, and eventually yield classification results. The convolutional layers are responsible for extracting features, while the linear layers are utilized for result prediction. Consequently, the feature extraction component can be regarded as the encoder, converting the original image into an intermediate representation suitable for machine learning, and the decoder transforms the intermediate representation into the desired output. The encoder-decoder architecture based on CNN is particularly useful for image processing tasks. For instance, Yasrab et al. ³⁶ proposed a simplified CNN-based encoder-decoder model for semantic segmentation. The encoder network is based on the twelve convolutional layers of VGG-16, omitting pooling layers, while the decoder network employs upsampling and transpose convolutional units, culminating in a pixel-level classification layer. Additionally, Badrinarayanan et al. ³⁷ introduced a practical fully convolutional neural network framework based on the encoder-decoder architecture. The encoding component has an analogous topological structure to VGG-16, while the decoder section utilizes a novel upsampling technique. Specifically, the decoder employs the pooling indices computed during the corresponding encoder’s max pooling steps to accomplish non-linear upsampling. Moreover, the proposed model, DeepLabv3+, ³⁸ expands on DeepLabv3 by introducing a simple yet effective decoder module to refine segmentation results, particularly at object boundaries. Furthermore, Li et al. ³⁹ achieved semantic segmentation tasks with minimal data by utilizing a CNN-based encoder-decoder model, enabling training with small sample sizes.

The proposed Micro-KTNet

System overview

In this study, we define the task as the recognition of the microstructure aggregation of fiber parent particles at the pixel level in SEM images. As depicted in Figure 1, our proposed framework comprises three main components: (a) transfer Learning for microstructure features (b) the SENetEncoder, and (c) the UnetDecoder.

Figure 1.

The proposed method, the Micro-KTNet Encoder-Decoder network, comprises three key components: transfer learning microstructure features, SENetEncoder, and UNetDecoder. The transferable microstructural features are used to initialize the proposed network. The SENetEncoder reuses convolutional layers trained on MicroNet to encode the input image into feature maps. The UNetDecoder is responsible for decoding these feature maps. Finally, the SegmentationHead processes the decoded feature maps to obtain per-pixel class probabilities, generating the segmentation mask.

The complete structure of Micro-KTNet, comprising its three key modules (the transfer learning microstructure features, the SENetEncoder module, and the UNetDecoder module), is illustrated, with detailed information regarding the training parameters and the learning process for the Micro-KTNet network provided in Section.

Traditionally, the weights of a neural network are initialized randomly. However, this may lead to overfitting problems that occur when the model fits too closely to the training data and fail to generalize effectively to new data. This challenge is more likely in the domain of fiber parent particle microstructure aggregation, where data is scarce and creating segmentation masks is costly. To address this, we adopt a transfer learning approach to initialize our network. Initially, a batch of pre-processed images, along with their corresponding ground truth, are fed into the model. The SENetEncoder is then employed to encode these inputs into feature maps with various resolutions, which contain valuable semantic information. Subsequently, the UnetDecoder is used to upsample these feature maps. The segmentation head is applied to the upsampled output to generate a segmentation mask. This mask contains class probabilities for each pixel, facilitating the identification of fiber parent particle microstructure aggregation. Eventually, the network delivers classification results for each pixel. Throughout the learning process, the segmentation network extracts feature information related to fiber parent particle microstructure aggregation from input images through forward propagation. The loss computation occurs during forward propagation, and network parameters are adjusted through backward propagation to minimize this loss, thereby optimizing the network’s performance in pixel-wise segmentation tasks. Ultimately, the output of the segmentation network precisely segments fiber parent particle microstructure aggregation, enabling automated analysis of specific structures within the images.

Transfer learning-based microstructure features

Collecting and manually creating segmentation masks can pose challenges and be costly, resulting in a relatively small self-made dataset. To mitigate the risk of overfitting, we utilize transfer learning to initialize our network. Initially, we train a ResNeXt50 network for classification tasks using a large microscope dataset. The objective is to learn a target function $f_{\theta }$ on the given source domain $D_{source}$ , by training a set of parameters $\theta *$ according to the following formula (4)

${\theta }^{*}=\mathit{arg}\underset{\theta }{\mathit{min}} L\left(D_{\mathit{source}};\theta \right)$

(4)where $\theta *$ represents the optimal model parameters. $\theta$ represents the parameters of the function, and L denotes the loss function. Previous studies have shown that the early layers of a network learn generic features, while the deeper layers focus more on specific features. By utilizing a pre-trained model on the large MicroNet microscope dataset, ³⁹ we enable our model to learn to detect higher-level microscopic structural features. Therefore, we choose all the convolutional layers of the ResNeXt50 network for transfer learning. Let the parameters of the SENetEncoder part be θ₁, which is equivalent to θ. The parameters of the UnetDecoder part are denoted as θ₂. In the target domain $D_{target}$ , we learn a target function $f_{\theta }$ by training a set of parameters $\left({\theta }_1{,\theta }_2\right)$ as per the following equation (5)

${\theta }^{*}=\mathit{arg}\underset{\left({\theta }_1,{\theta }_2\right)}{\mathit{min}} L\left(\left({\theta }_1,{\theta }_2\right),D_{\mathit{target}}\right)$

(5)where $\theta *$ represents the optimal model parameters, and θ₂ is randomly initialized parameters. The loss function $L(·)$ used consists of a weighted combination of the BCEloss (Binary Cross-Entropy) and the Dice loss. Let the predicted binary segmentation mask be P and the true binary segmentation mask be G. The BCEloss (Binary Cross-Entropy) is defined as formula (6)

$BCELoss\left(P,G,\theta \right)=-\frac{1}{N}{\displaystyle \sum }_{i=1} \left[G_i\times \log \left(f_{i|\theta }\right)+\left(1-G_i\right)\times \log \left(1-f_{i|\theta }\right)\right]$

(6)where N represents the number of pixels, G _i is the value of the $i-th$ pixel in the true binary segmentation mask (0 or 1), and $f_{i|\theta }$ is the corresponding predicted probability. The Dice loss is defined as formula (7)

$\mathit{DiceLoss}\left(P,G,\theta \right)=1-\frac{2\sum N_{i=1}{P}_i\times G_i}{\sum N_{i=1}{P}_i^2+\sum N_{i=1}{G}_i^2}$

(7)

The overall loss function of transfer learning is defined as formula (8)

$\mathit{Loss}\left(\theta \right)=\partial \times \mathit{BCELoss}\hspace{0.17em}\left(P,G,\theta \right)+(1-\partial )\times \mathit{DiceLoss}\left(P,G,\theta \right)$

(8)where ∂ is a hyperparameter that controls the weighting between the BCE loss and the Dice loss.

SENetEncoder module for few-labeled SEM images

Due to lacking the labeled SEM images of masterbatch agglomeration microstructure, we repurpose the convolutional layers of the ResNeXt50 network as part of the SeNetBlock. Figure 2 illustrates the structure of the SeNetBlock with downsampling, which comprises several processing blocks. The input image goes through convolutional layers, batch normalization layers and nonlinear activation layers to generate feature maps as formulas (9)–(11)

$U_{cj}^I={\displaystyle \sum }_{i\in M_j} X_{cj}^{I-1}*K_{ij}^I$

(9)

$O_{cj}^I=\frac{U_{cj}^I-{\mu }_{cj}^I}{\sqrt{{\sigma }_{cj}^I+ϵ}}·{\gamma }_{cj}^I+{\beta }_{cj}^I$

(10)

$X_{cj}^I=f\left(O_{cj}^I\right)$

(11)where f (·) represents the ReLU activation function. X ^I−1_{c j} denotes the $j-th$ channel of the feature map at the $\left(I-1\right)-th$ layer. $K_i^I$ represents the convolutional kernel. M_j signifies the input subset, and $‘*\text{'}$ denotes the convolution operation. $U_{cj}^I$ represents the network activation after the I-th layer’s convolution operation. As shown in Figure 2, the first processing block consists of a 1 × 1 convolution (Conv1), a batch normalization layer (bn1), and a nonlinear activation layer (ReLU). The second processing block includes a 3 × 3 convolution (Conv2), a batch normalization layer, and a nonlinear activation layer. For the second processing block, the input $X^2$ goes through a 1 × 1 convolution layer (Conv3) and a batch normalization layer to obtain O³. O³ is then divided into two parts, with one part passing through the SEModule. The SEModule is composed of a pooling layer, a 1 × 1 convolutional layer, and a nonlinear activation layer. Additionally, O³ undergoes average pooling, a 1 × 1 convolution (FC1), a rectified linear unit, a further 1 × 1 convolution (FC2), and finally the sigmoid function. The output is X³, which is obtained by element-wise multiplication with X³, and at the same time, the input goes through a layer of 1 × 1 convolution (Conv0) and batch normalization layer to obtain O⁴

$O^4=f_{bn}\left(\text{Conv}\left(X^0\right)\right)$

(12)

$X^4=f_{\text{relu}}\left(O^4+X^3\odot O^3\right)$

(13)where X⁰ is the input of the SeNetBlock, $\odot$ denotes element wise multiplication, and $+$ denotes element-wise addition. The output X⁴∈ $R^{k_l\times F_l\times T_l}$ of the fourth layer is the output of the SeNetBlock. Similarly, for the SeNetBlock without a downsampling module, $O^4$ is the input data.

Figure 2.

The diagram illustrates the structure of the SeNetBlock with Downsampling.

Figure 2 illustrates the hierarchical structure of the encoder path, which comprises five layers. Each layer takes as input from either the original image or the encoding feature map from the previous layer. The output of each layer is both forwarded to the next layer and directly transmitted to the decoder through skip connections. The initial layer of the encoder consists of a convolutional layer with a 7 × 7 kernel size, a stride of 2, and a padding size of 3. This convolutional layer’s primary purpose is to extract initial features, which are subsequently subjected to batch normalization and ReLU activation before being downsampled by a max pooling layer. This operation reduces the feature map’s size by half while keeping the number of channels unchanged. The first SeNetBlock block in the encoder is redesigned for downsampling feature information. Each downsampling step diminishes the feature map’s size by half while doubling the number of channels. The encoder progressively reduces resolution through convolution and pooling operations, generating multi-level features with varying resolutions. These features are subsequently transmitted to the decoder via skip connections. In contrast, the decoder gradually increases resolution through operations such as transposed convolution. The inclusion of skip connections serves two key purposes: firstly, it enables high-resolution feature information to flow seamlessly between the downsampling and upsampling processes. Secondly, skip connections aid in preserving finer details when handling small targets or boundaries.

UnetDecoder module for microstructure features

Our decoder comprises four layers, with each layer receiving the encoding feature maps from its corresponding layer and higher layers. The fourth layer of the decoder is taken as an illustrative example: the encoding feature map from the fifth layer undergoes an initial upsampling process, increasing the number of feature maps while reducing their size. This upsampled feature map, achieved through transposed convolution, matches in size with the encoding feature map of the fourth layer

$F_{\mathit{upsampled}}\left[i,j,c\right]={\displaystyle \sum }_{m,n} F_{{\mathit{trans}}_{\mathit{conv}}}\left[i-m,j-n,c,k\right]·F_{{\mathit{cn}}_{\mathit{fin}}}\left[m,n,k\right]$

(14)where ${F_{trans}}_{conv}$ represents the weight of the transposed convolution, ‘·’ denotes element-wise multiplication. After performing transposed convolution, the size of the obtained feature map $F_{upsampled}$ is $i_{*}j$ with a channel number of c. Next, the obtained feature map $F_{upsampled}$ is concatenated with the feature map F2 directly passed from the fourth layer of the encoder

$F_{\text{upsampled}}\left[i,j,c\right]=\{\begin{array}{l}{F}_{\text{upsampled}}\left[i,j,c\right],if\hspace{0.17em}o\le k<C_1\\ F_{\text{upsampled}}\left[i,j,c\right],if\hspace{0.17em}{C}_1\le k<C_1+C_2\end{array}$

(15)

The channel number of the concatenated feature map $F_{concat}$ , formed by combining feature maps C₁ and C₂, is C₁+C₂. The concatenated feature map then undergoes an attention module, followed by two convolutional layers. Subsequently, a second attention module is applied, followed by another upsampling step to align the feature map size with that of the encoding feature map from the third layer. Within the upsampling pathway, the number of feature maps gradually decreases, while their spatial dimensions progressively increase. Moreover, the scSE attention module ⁴⁰ was incorporated to enhance our model’s understanding of both local and global image structures. The scSE attention can be expressed as two parallel modules: the s SE module (Spatial Squeeze module) and the cSE module (Channel Squeeze module). The overall architecture of the scSE attention module is depicted in Figure 3. In the c SE module, the research ⁴⁰ assumes that the input feature map is a combination of channels U_i ∈ R ^H×W. The channel information is compressed using global average pooling to produce a feature vector Z ∈ R ^{1×1 ×C}, where the feature vector of the k-th layer is given by formula (16)

$Z_k=\frac{1}{H\times W}{\displaystyle \sum }_i {\displaystyle \sum }_j W_j{u}_k\left(i,j\right)$

(16)where the vector z contains global spatial information. Next, the vector passes through two fully connected (FC)layers and a ReLU operator $\sigma$ . The resulting vector is used for recalibration or activation of U as formula (17)

${\widehat{U}}_{cSE}=F_{cSE}\left(U\right)=\left[\sigma \left({\widehat{Z}}_1\right)u_1,\sigma \left({\widehat{Z}}_2\right)u_2,\dots ,\sigma \left({\widehat{Z}}_C\right)u_C\right]$

(17)where the activation $\sigma \left(\stackrel{\sim }{Z_i}\right)$ represents the importance of the i-th channel. In the SSE module, ⁴⁰ introduced the spatial squeeze operation is achieved through a convolution $q\hspace{0.17em}{{=W}_{sq}}_{*}U$ , where the weight $W_{sq\hspace{0.17em}}\in {\hspace{0.17em}R}^{1\times 1\times C\times 1}$ , resulting in a projection tensor q ∈ R ^H×W. The generated vector is then used for recalibration or activation of U as formula (18)

${\widehat{U}}_{SSE}=F_{SSE}\left(U\right)=[\sigma \left(q_{1,1}\right)u^{1,1},\dots ,\sigma \left(q_{i,j}\right)u^{i,j},\dots ,\sigma \left(q_{H,W}\right)u^{H,W}]$

(18)where each value σ (q_{i, j}) corresponds to the relative importance of the spatial information (i, j) in the given feature map. As shown in Figure 3, research ⁴⁰ obtain the concurrent spatial and channel SE, $\stackrel{\sim }{U_{SCSE}}$ , by element-wise addition of the channel and spatial excitation, $\stackrel{\sim }{U_{SCSE}}=\stackrel{\sim }{U_{CSE}}+\stackrel{\sim }{U_{SSE}}$ . This step integrates information from both spatial and channel perspectives, enabling the model to focus more effectively on crucial spatial locations and channel features.

Figure 3.

The diagram depicts the structure of the scSE attention module.

Learning the proposed network

A network initialized with transfer learning requires only a small amount of training data. In this study, the Micro-KTNet model was trained using a small dataset of fiber granule aggregation and its performance was evaluated on test samples. During the training process, the Adam50 optimizer was employed with an initial learning rate of 2e-4. The loss function used was a weighted combination of the balanced cross-entropy (BCE) and Dice loss, with BCE assigned a weight of 70%. BCE (Binary Cross-Entropy) is generally more stable during the initial phases of training, which aids in speeding up model convergence. On the other hand, Dice Loss places greater emphasis on pixel-level similarity and promotes the generation of smoother and more continuous segmentation results. By combining BCE with Dice Loss, a reduction in overall loss was achieved while still benefiting from the stability conferred by BCE. The implementation of the Micro-KTNet model was carried out using PyTorch, the segmentation models library, and other relevant third-party libraries. These experiments were conducted on a personal computer with 24 GB of memory and an Nvidia GeForce RTX 3090 graphics processing unit, utilizing Python for implementation.

Experimental evaluation and discussion

In this study, the performance of the Micro-KTNet model was assessed through a series of experiments focused on segmenting fiber granule microstructure aggregation. The following sections provide detailed explanations of the experimental setup and the implementation of the Micro-KTNet network. Furthermore, its performance was compared with other algorithms designed for segmenting fiber granule microstructure aggregation. Finally, an ablation study was conducted to dissect various components of the Micro-KTNet model.

Data creation and evaluation metrics

A scanning electron microscope (SEM) was used to capture high-resolution microstructure images of fiber granules. These images are detailed, but they also contain some unwanted interference, such as stains. To mitigate these issues stemming from the acquisition process, a series of preprocessing steps involving transformation, segmentation, and diversification techniques were employed. This culminated in the creation of a dataset termed SEM DHU-150, consisting of microstructure aggregation data from fiber granules. In SEM-DHU-150, positive labels denote particles resulting from aggregation in fiber granule microstructures, while negative labels are assigned to other structures, such as gaps between fibers.

To construct the dataset, image patches of size 512 × 512 were randomly extracted from the high-resolution original images. The segmentation results of the original images are demonstrated in Figure 4. Positive samples were manually labeled. Ultimately, around 150 samples were assembled for the dataset, with 90 samples allocated for the training set, 40 samples for validation, and the remaining samples earmarked for prediction. Given the limited number of training set samples, data augmentation techniques were adopted to enhance the model’s performance. Firstly, random horizontal flips were applied to the images with a 30% probability to make the model learn invariant image features. Additionally, random adjustments to contrast, brightness, and gamma were introduced to increase data diversity and enhance the model’s adaptability to variations in lighting conditions. Lastly, blurring and image sharpening effects were incorporated, prompting the model to focus more on critical features, such as image edges, during training. The segmentation results of the original images are demonstrated in Figure 5. The results of data augmentation are shown in Figure 6.

Figure 4.

An example illustrating the segmentation process of an original image.

Figure 5.

The result of data augmentation: Images (a) and (e) are the original unprocessed images, while images (b) and (f) represent the modified versions with a horizontal flip, (c) and (g) represent adjustments to brightness and contrast respectively, (d) and (h) are undergoing blurring and sharpening processes.

Figure 6.

Figure (a–f) showcase examples of fiber masterbatch aggregation segmentation. In these examples, yellow pixels represent true positives, green pixels represent false negatives, red pixels represent false positives, and blue pixels represent true negatives.

To evaluate the effectiveness of the Micro-KTNet model, four quantitative metrics were employed: IOU, precision, recall, and F1-Score, which are defined as follows

$IOU=\frac{TP}{TP+FN+FP}\times 100\%$

(19)

$\text{precision}=\frac{TP}{TP+FP}\times 100\%$

(20)

$\text{recall}=\frac{TP}{TP+FN}\times 100\%$

(21)

$F1\text{score}=2\times \frac{\text{precision}\times \text{recall}}{\text{precision}+\text{recall}}\times 100\%$

(22)where TP and FN refer to the proportions of samples identified as having clustering and not having clustering, respectively. FP refers to the proportion of samples not having clustering that are incorrectly identified as having clustering.

Experiment results

In this study, the effectiveness of the fiber granule microstructure segmentation method was assessed and a comparative analysis of its segmentation results was conducted against other methodologies.

The Micro-KTNet model demonstrates impressive performance in segmenting fiber granule aggregates. When dealing with SEM images containing only large-sized aggregate particles within fiber granules, the proposed network accurately segments these particles. Even in scenarios where SEM images contain multiple aggregates with slight variations in size, our network still performs admirably. However, challenges arise when SEM images feature a substantial number of aggregate particles with considerable size differences, leading to occasional mis-segmentation or missed segmentation instances. This observation is evident in Figure 6(a) and (e), and it could be attributed to two key factors. Firstly, some aggregate particles may be situated at the edges of the image, where the model lacks adequate contextual information to effectively learn these features, resulting in instances of missed segmentation, as depicted in Figure 6(a). Secondly, the complexity of the fiber texture can complicate the differentiation process, as certain areas of the fiber texture may resemble the texture of small-sized aggregates. This similarity makes it challenging for the model to distinguish between them accurately. In summary, this model achieves an IOU (Intersection over Union) score of 86. 65% on the SEM-DHU-150 dataset, signifying its overall strong performance in segmenting fiber granule microstructures.

Comparative analysis

In the evaluation on the SEM-DHU-150 dataset, the segmentation results produced by Micro-KTNet were compared with several commonly used segmentation methods. Among the chosen approaches, Lin et al. ⁴¹ leverages a multi scale feature pyramid to integrate semantic information from different levels. Zhao et al. ⁴² employs a structure called Pyramid Pooling to capture multi-scale contextual information. Additionally, Unet, ⁴³ LinkNet, ⁴⁴ MANet, ⁴⁵ UnetPlusPlus, ⁴⁶ and DeepLabV3Plus ³⁸ all follow an Encoder-Decoder architecture. Unet distinguishes itself with skip connections, while MANet introduces attention mechanisms based on the Unet framework. UnetPlusPlus extends network depth and adopts a more densely connected structure. DeepLabV3 ⁴⁷ employs dilated convolutions to expand the receptive field, along with global average pooling and multi-scale information fusion to enhance object boundary segmentation. DeepLabV3Plus further enhances segmentation accuracy by adopting an encoder-decoder structure.

The quantitative results presented in Table 1 demonstrate that Micro-KTNet outperforms all other methods in terms of IOU (Intersection over Union). This outcome aligns with our initial expectations, suggesting that the transfer learning approach enhances the model’s generalization performance. As shown in Figure 7, the proposed framework indeed achieves the highest F1-score compared with other methods. This achievement can be partly attributed to the attention module in the decoder, which integrates both global and local information, resulting in a more balanced precision and recall. Ablation experiments were conducted in Section 4.4 to further investigate the effectiveness of these two modules. Additionally, a qualitative analysis is conducted on partial segmentation results of selected networks, as shown in Figure 8.

Table 1.

The performance of different models on the fiber masterbatch microstructures aggregation dataset.

Methods	IOU (%)	Precision (%)	Recall (%)	F1score (%)	Test time (ms)
PSPnet	62.43	57.46	64.43	63.62	4.43
FPN	64.84	68.95	68.60	66.27	6.54
LinkNet	72.12	70.25	77.23	74.44	6.76
MANet	75.22	74.05	77.21	75.62	12.04
DeepLabV3	74.68	81.76	82.32	81.10	13.24
Unet	74.05	78.35	81.20	80.31	7.16
UnetPlusPlus	76.00	82.43	88.39	84.35	15.82
Micro-KTNet	86.65	91.85	97.21	94.50	8.94

Bold values represent the best statistically significant results.

Figure 7.

The F1-score of each method during learning procedure.

Figure 8.

The segmentation results of different models on the fiber masterbatch super microstructure aggregation dataset. (a) to (e) show the results of aggregate particle size decreasing in the input images, where (a) exhibits the largest aggregated particles and (e) the smallest. In these results, yellow pixels represent true positives, green pixels represent false negatives, red pixels represent false positives, and blue pixels represent true negatives.

When the input consists of Figure 8(d) and (e), both DeepLabV3 and Unet tend to miss the desired parts of the aggregate particles (indicated in green). Particularly, Unet struggles with segmenting the edges, and its performance decreases significantly when the input is Figure 8(c), where nearly 60%of the aggregate particles are not properly segmented. UnetPlusPlus, while an improvement over Unet, still suffers from issues such as segmenting out undesired aggregate particles at the image edges (highlighted in red). Comparatively, the proposed Micro-KTNet network exhibits fewer misidentifications at the image edges and significantly outperforms both DeepLabV3 and Unet in segmenting the desired aggregate particles. In fact, the Micro-KTNet network achieves nearly perfect segmentation in these cases. In summary, the proposed Micro-KTNet network demonstrates the most robust performance in binary segmentation, outperforming the other methods considered in benchmark dataset of masterbatch agglomeration.

Ablation experiments

In this section, the strong performance of the Micro-KTNet model was attributed to the enhancements in model generalization achieved through transfer learning and the incorporation of the attention module in the decoder. This section delves into the effectiveness of these two components through ablation experiments. Table 2 presents the ablation results obtained when either the transfer learning component or the attention module is excluded. The ablation study reveals that the segmentation performance of Micro-KTNet (without the transfer learning component) is inferior to that of Micro-KTNet (without the attention module). This indicates that the transfer learning component plays a more pivotal role in the Micro-KTNet.

Table 2.

The performance of the Micro-KTNet model on the fiber masterbatch microstructure aggregation dataset under different conditions.

Methods	IOU (%)
Micro-KTNet (without attention module and transfer learning component)	72.12
Micro-KTNet (without attention module)	83.18
Micro-KTNet (without transfer learning component)	75.37
Micro-KTNet-MicroNet	86.65
Micro-KTNet-ImageNet	85.07
Micro-KTNet-Image-MicroNet	87.04

Bold values represent the best statistically significant results.

Table 2 also sheds light on the impact of reusing convolutional layers pre-trained on various datasets, including the MicroNet dataset, the ImageNet dataset, and a combined Image-MicroNet dataset, on the model’s segmentation performance. Remarkably, the model performs exceptionally well in all three cases. However, it’s noteworthy that the model pre-trained on the MicroNet dataset outperforms the one trained on the ImageNet dataset. One plausible explanation is that the filters trained on the ImageNet dataset may not be entirely suited for the unique characteristics of fiber mother grain nanostructure aggregation images. Nevertheless, the optimal approach appears to be pre-training the convolutional neural network on ImageNet and subsequently fine-tuning it on the MicroNet dataset.

Assessing the impact of architectures

In this study, A large number of experiments were conducted to explore the effects of replacing ResNeXt-50 with various popular network architectures. These architectures include DenseNet121, ²⁴ DPN107, ⁴⁸ EfficientNet-B5, ⁴⁹ Inception-ResNet-V2, ⁵⁰ Inception-V4, ⁵¹ MobileNet-V2, ⁵² ResNet-50, ⁵³ ResNeXt-50_32 × 4d, ⁵⁴ SENet-154, ⁵⁵ VGG-16_bn, ⁵⁶ and Xception. ⁵⁷ All networks were fine-tuned using transfer learning, with the same training configurations applied to ensure consistency. The results showed notable differences in performance. The IOU (Intersection over Union) scores ranged from 66.01% (MobileNet-V2) to 86.65% (ResNeXt-50_32 × 4d).

Table 3 showed that ResNeXt-50_32 × 4d achieved the highest IOU of 86.65%. This result highlights its strong ability to extract features for this task. SENet-154 and Xception also performed well, with IOU scores of 85.08% and 85.17%, respectively. This suggests that models with advanced feature integration mechanisms can perform better. In contrast, EfficientNet-B5 and MobileNet-V2 had lower scores, with IOUs of 67.95% and 66.01%. Their lightweight designs may have limited their feature extraction capabilities. Models like ResNet-50, VGG-16_bn, and the Inception series (Inception-ResNet-V2 and Inception-V4) showed moderate performance. These results confirm that the choice of encoder architecture has a significant impact on task performance.

Table 3.

The performance of the Micro-KTNet model on the fiber masterbatch microstructure aggregation dataset with different encoder.

Methods	IOU (%)
DenseNet121	82.48
dpn107	83.97
EfficientNet-b5	67.95
Inception-ResNet-V2	81.37
Inception-V4	77.76
MobileNet-V2	66.01
ResNet-50	82.30
ResNeXt-50_32 × 4d	86.65
SENet-154	85.08
VGG-16_bn	81.00
Xception	85.17

Bold values represent the best statistically significant results.

Assessing the impact of hyperparameters

In this experiment, some hyperparameters, such as learning rate and batch size, were found to affect the performance of the proposed method. To explore the relationship between hyperparameters and segmentation results, four experiments with different batch sizes were conducted, and the model’s performance was evaluated using four metrics (IOU, F1-score, precision, recall), as shown in Figure 9. A larger batch size enables the model to consider more samples in each update step, facilitating the learning of complex patterns and structures in the data. For example, when the batch size increased from 2 to 14 in the experiment, the IOU improved from 81.12% to 86.59%, the F1 score rose from 86.93% to 94.50%, and the precision and recall increased from 82.33% and 88.31% to 91.85% and 97.21%, respectively. These improvements demonstrate how a larger batch size can stabilize the gradient updates and enhance the model’s ability to generalize patterns within the data.

Figure 9.

Evaluation of performance with different batch size.

However, when the batch size became too large, such as increasing it to 20, the metrics began to decline. For instance, the IOU dropped to 81.72%, the F1 score decreased to 87.03%, and both precision and recall fell to 82.56% and 89.13%, respectively. This decline suggests that while larger batch sizes allow the model to process more data in a single iteration, they also reduce the number of gradient update steps within an epoch. This can lead to less frequent weight updates and the loss of fine-grained gradient information, which is critical for learning subtle patterns in the data.

Furthermore, in the context of smaller datasets, this problem can be exacerbated. Excessively large batch sizes mean that the model might process the entire dataset in just a few iterations, limiting its ability to fully explore the data distribution and learn diverse features. As seen in this case, when the batch size continued to increase to 26 and 32 (hypothetical data), the IOU further decreased to 81.00% and 79.80%, while the F1 score fell to 86.00% and 84.20%. These trends illustrate the trade-off: while larger batch sizes improve computational efficiency, they can hurt the model’s generalization ability if set excessively high.

Assessing the impact of hyperparameters

The purpose of this experiment was to investigate the impact of the number of training samples on the performance of the model, providing further validation of its adaptability and generalization ability under limited data conditions. By adjusting the size of the training set, the study aimed to evaluate the model’s performance in scenarios with extremely few labeled samples and reveal its behavior as the amount of data increases.

In the experimental setup, the test set was fixed at 40 images, while the training set was divided into subsets with 1 image (40:1), 5 images (8:1), 10 images (4:1), 20 images (2:1), and 40 images (1:1). The model’s performance was evaluated using the Intersection over Union (IoU) metric as the primary indicator, complemented by pixel-level classifications illustrated in the provided visual results (Figure 10).

Figure 10.

(a) shows an example from the test set of the SEM-DHU-150 dataset. (b) shows the evaluation of performance with different numbers of training samples. In these results, yellow pixels represent true positives, green pixels represent false negatives, red pixels represent false positives, and blue pixels represent true negatives.

The IoU results demonstrated that the model achieved 79.55% IoU with 40 training samples, 75.61% IoU with 20 samples, 68.59% IoU with 10 samples, 46.32% IoU with 5 samples, and 23.74% IoU with only 1 sample. Several observations can be drawn from these results. First, even under conditions with very few samples, such as 5 images (46.32% IoU), the model still exhibited relatively reasonable performance. This is attributed to the use of a pre-trained encoder, which was trained on Micronet and possesses strong feature extraction capabilities, thereby enabling effective performance with limited training data. Second, as the number of training samples increased, the model’s performance improved steadily. For example, IoU increased from 23.74% with 1 image to 79.55% with 40 images. This improvement is due to the additional training data enabling the model to enhance its recognition and generalization ability when the ratio of training to testing data is below 1:1. Third, in the most extreme case, with only 1 training image, the results were suboptimal. This is likely because the information contained in a single image is significantly less representative compared to the 40-image test set, causing the model to focus excessively on limited features and struggle with generalization.

The results of this experiment provide further evidence of the model’s strong performance in low-resource scenarios while also highlighting the advantages of increasing training data for achieving optimal performance. Under extremely limited data conditions, the results depend heavily on the quality of the pre-trained features, as demonstrated in the 5-training scenario, where the pre-trained encoder contributed significantly to the relatively strong performance.

Conclusions and future work

This paper proposes a novel microstructure recognition architecture, named Micro-KTNet, which is designed for segmenting SEM images of fiber masterbatch aggregation. The model incorporates a transfer learning-based encoder, pre-trained on the MicroNet dataset, to extract domain-specific features effectively. The decoder, enhanced with SCSE attention modules, combines global and local features to achieve precise segmentation. The encoder-decoder architecture with skip connections facilitates multi-scale feature utilization, ensuring detailed and contextually consistent predictions. On the SEM-DHU-150 dataset, Micro-KTNet achieves state-of-the-art performance, with an IOU of 86.65% and an F1-score of 94.50%. Despite its strong performance, challenges remain in segmenting aggregation particles near image edges and in distinguishing fine textures within complex regions. Future work will address these limitations while also focusing on integrating explainable AI methods, such as visualizing attention maps, to enhance the transparency of the model’s decision-making process. This addition will improve industry trust and facilitate broader adoption in industrial environments.