Advancing crack detection with generative AI for structural health monitoring

Abstract

Monitoring structural integrity through accurate crack detection is fundamental to ensuring the safety and longevity of civil engineering infrastructure. Vision-based methods, supported by advancements in deep learning, have gained prominence in structural health monitoring (SHM). However, these methods often suffer from limited performance due to insufficient diversity and scale in crack image datasets, which are costly and challenging to acquire. This study introduces a novel framework that integrates a text-to-image generative model with large language models to synthesize realistic crack images for training deep neural networks. A prompt engineering approach is utilized to generate high-quality textual descriptions, guiding the creation of a large-scale, diverse dataset that simulates a wide range of crack scenarios. The synthesized dataset significantly enhances model training, as demonstrated in two key SHM tasks: crack classification and crack object detection. Neural networks trained with the augmented dataset show up to a 60% improvement in precision over baseline models trained on real-world data alone. These results highlight the potential of generative models to address data scarcity in SHM, enabling more robust and accurate crack detection. This research provides a scalable and efficient solution for improving machine learning-based SHM applications and paves the way for further exploration of generative methods in structural monitoring tasks.

Introduction

Regular assessments are essential to maintain the safety of civil structures and safeguard these critical assets. A key aspect of these evaluations is the detection of cracks, which is often an early indicator of potential structural issues.^1,2 Traditional visual inspection methods for detecting cracks are ineffective, time consuming, require significant manpower and are prone to human error and subjective conclusions. These methods are also costly, and often struggle with accessibility issues, especially in large-scale, long-span and high-rise structures.

Advancements in computer vision (CV) technologies have led to a shift towards using image-based methods for crack detection in structural health monitoring (SHM). These methods, which capture digital images of structures and analyse them with software to detect cracks, provide a faster, more cost-effective, and robust alternative to traditional inspection methods. Vision-based crack detection methods primarily fall into two categories: image processing and machine learning (ML). Image processing techniques employ hand crafted filters, ³ morphological operations ⁴ and percolation theories, ⁵ among other techniques, to detect cracks without the necessity for training a model. Over the past decade, ML has emerged as a dominant technique in CV-based crack detection, ⁶ known for its exceptional performance across various tasks and gaining popularity in numerous fields.^7–10 When provided with sufficient data, ML algorithms can autonomously identify complex features and hidden patterns, making them highly effective for precision-critical tasks, such as crack detection.

The task of crack detection within ML can be divided into three primary functions: crack image classification, crack object detection and crack segmentation. The goal of crack image classification^11–13 is to ascertain whether an image or image patch contains cracks. A notable study by Gopalakrishnan et al. ¹³ utilized a Visual Geometry Group (VGG) model ¹⁴ to detect cracks on hot-mix asphalt and Portland cement concrete surfaces, achieving successful classification of pavement crack images. Different from crack image classification, crack object detection tasks^15–19 involve generating bounding boxes around regions identified as containing cracks. Currently, prominent object detection models like Fast Region-based Convolutional Network ²⁰ and Single Shot MultiBox Detector ²¹ are employed for this task. Crack segmentation^22–24 provides the most detailed, pixel-level detection of images, classifying each pixel as either part of a crack or not. Among these methods, convolutional neural networks (CNNs) are popular for crack segmentation. Schmugge et al. ²⁵ developed a CNN-based technique to segment cracks in nuclear power plant inspections by aggregating pixel-level classification confidence across video frames under different lighting conditions. Islam and Kim ²⁶ developed a crack segmentation system using a fully convolutional network (FCN) with a VGG backbone. Trained on an open-source concrete crack dataset, the optimized FCN achieved around 92% in both accuracy and F1 scores on test data.

Despite the rapid advancements in ML methods for crack detection, the scale and diversity of training datasets remain limited. In the realm of ML, contemporary research increasingly supports that diverse large-scale datasets significantly enhance the robustness and effectiveness of Deep Neural Networks (DNNs). For instance, the popular deep learning model Segment Anything ²⁷ used extensive datasets to enhance its segmentation capabilities, showing that with ample data, DNNs can achieve high-precision segmentation in various scenarios and adapt effectively to new environments. In the field of ML-based crack detection, there is a keen interest in gathering more crack images to train ML models. This demand stems from the need to train models that can accurately reflect the wide variability in crack appearances, which depend heavily on factors such as the material of structures, the types of damage they have sustained and environmental influences. However, gathering diverse crack images is challenging due to restricted access, high costs and environmental constraints. Restricted access means that safety regulations or property rights often limit entry to damaged structures. High costs arise from the need for specialized equipment and trained personnel. Environmental constraints include challenges such as lighting conditions and seasonal variations, which can affect the visibility and detectability of cracks. These challenges underscore the difficulties in collecting real crack images and the urgent need to develop methods for acquiring large-scale, diverse datasets. Consequently, enhancing existing datasets with synthesized data has become a promising area of research.

Many studies have explored the creation of synthesized datasets for training neural networks. Among these methods, physics-based approaches^28,29 employed intricate mathematical models to simulate the physical properties and behaviours of real-world structures. This often involves creating three-dimensional models of structures and then applying simulated forces (e.g., wind, earthquakes) to assess how these structures might react. Conversely, data-driven approaches, particularly generative adversarial networks (GANs), ³⁰ have been employed to create visually realistic image datasets for SHM tasks.^31,32 For example, GANs have been utilized to generate synthesized data for wind turbine fault detection to enhance fault detection, ³³ and for railway crack detection. ³⁴ Both physics-based methods and GANs present innovative approaches to synthesized dataset generation, yet each exhibits distinct limitations. Physics-based models produce data with high physical realism and require extensive domain knowledge and significant computational resources, which can limit their scalability and practical application in large-scale dataset generation. GANs have demonstrated success in producing realistic visual patterns; however, ensuring consistent performance across highly diverse structural scenarios can be challenging, and careful training is often needed to maintain image fidelity and diversity.

In response to the challenges, this article proposes a framework for synthesized crack image dataset generation that leverages a diffusion model in combination with a large language model (LLM). Diffusion models have emerged as a powerful tool for generating high-quality and diverse images from textual prompts. Compared to GANs, diffusion models offer improved controllability and variation in output, as they can be guided by richly descriptive prompts to generate images that align with specific structural contexts. Furthermore, modern diffusion models are typically large-scale, pretrained on vast and diverse datasets. This broad training foundation equips them with strong generalization capabilities, enabling them to synthesize realistic images even for unseen or complex scenarios. These advantages make diffusion models particularly suitable for applications such as SHM, where the ability to generate visually and contextually accurate damage scenarios is critical for training robust deep learning models. Furthermore, the integration of LLM accelerates the creation of varied and precise prompts, substantially increasing the diversity of the generated images. This approach outperforms traditional physics-based methods, producing 1K high-resolution, realistic images in just a few hours with a single GPU – an achievement unattainable with conventional methods. The efficacy of the synthesized data is evaluated through two experiments: crack image classification and crack object detection, using selected public datasets as benchmarks. By adding the synthesized images to the training set, the same DNN models become significantly more robust, achieving much better evaluation results. Crack image classification precision increased from 33.6 to 93.96%, and crack object detection precision improves from 24.25 to 83.82%.

Figure 1 illustrates a comparison between crack images from public crack datasets and synthesized crack images generated using the proposed framework. The synthesized images demonstrate a high level of realism, capturing various crack types, textures and environmental conditions that closely mimic those found in real-world scenarios.

Figure 1.

Comparison of crack images from public datasets (right) and synthesized crack images generated using the proposed framework (left).

The organization of this article is as follows: the second section elaborates the methodology of the data generation framework, detailing the integration of the diffusion model and an LLM for crafting synthesized images. The third section presents the methodologies employed and the results obtained for crack image classification and crack object detection. The fourth section provides a conclusion, summarizing the principal findings and suggesting for future research.

Synthesized crack image generation

Figure 2 presents a schematic overview of the synthesized crack image generation framework, employing a large text-to-image rectified flow model (RFM) ³⁵ that has been extensively trained on a vast dataset of billions of diverse text-image pairs, encompassing a wide array of scenes from urban environments to structural materials. Guided by textual prompts, the model can reconstruct images from noisy images to generate high-quality synthesized visualizations. To enrich the text prompts, the LLM Generative Pre-trained Transformer (GPT)-4 ³⁶ is employed to create varied and detailed text prompts. A crucial step involves a prompt engineering process that preprocesses raw text from GPT-4, selecting prompts that are preferred by the diffusion model for optimal results. These selected prompts are then processed by a text encoder, which converts the descriptions into a feature space representation that the diffusion model can utilize for image generation.

Figure 2.

Overview of the proposed synthesized crack image generation framework.

Text-to-image generation with RFM model

The diffusion process consists of two parts, one is forward diffusion, and the other is reverse diffusion. Forward diffusion is to add noise to the image, and reverse diffusion is to remove noise. Firstly, it is assumed that the image side is $x_0$ and the noise side is $x_T$ . In the forward diffusion process, noise is incrementally added to an image over a sequence of steps. Initially, the image, denoted as $x_0$ , is considered noise-free. As the process progresses, noise is added until the image, at step $T$ , denoted as $x_T$ , becomes predominantly noisy. Forward diffusion principle is expressed as:

$$q(x_t|x_{t-1})=N(x_t;\sqrt{1-{\beta }_t{x}_{t-1}},{\beta }_{t}I)$$

(1)

In the forward diffusion process, noise is added to an image incrementally through a sequence of steps. The transformation of the state variable $x_t$ at each time step $t$ is a transformation from the state variable $x_t$ , which follows a Gaussian distribution, to the state variable at the next moment, which follows another Gaussian distribution. The conditional distribution $q(x_t|x_{t-1})$ represents the Gaussian distribution of $x_t$ given the previous state variable $x_{t-1}$ . The distribution $q(x_t|x_{t-1})$ is defined as a Gaussian distribution with a mean of $\sqrt{1-{\beta }_t{x}_{t-1}}$ and a variance of ${\beta }_{t}I$ . Here, ${\beta }_t$ represents the variance parameter at step $t$ , and $I$ is the identity matrix, indicating that noise is added independently to each component of the image. The time variable $t$ ranges from 0 to $T$ , where $x_0$ is the initial, noise-free image and $x_T$ represents the image fully transformed by noise. The problem of forward diffusion is thus framed as determining the distribution of noise added at each step. This involves solving for the parameters of the Gaussian distributions that model the transition from $x_{t-1}$ to $x_t$ over the sequence from 0 to $T$ .

The process of adding noise to an image is defined as $z_t$ , and it is generally assumed to follow a normal distribution, denoted by $z~N(\mu ,{\sigma }^2)$ . This abstract process is expressed in the following form:

$$z=\mu +\sigma \epsilon$$

(2)

where $\epsilon ~N(0,1)$ . Similarly, the sample image at time $t$ , denoted as $x_t$ , is rewritten as:

$$x_t=\sqrt{1-{\beta }_t}{x}_{t-1}+\sqrt{{\beta }_t}{\epsilon }_{t-1}$$

(3)

Assume ${\alpha }_t=1-{\beta }_t$ , $\overline{{\alpha }_t}=\underset{i=1}{\overset{t}{\Pi }}{\alpha }_i$ , then Equation (3) can be recursively organized as follows:

$$x_t=\sqrt{\overline{{\alpha }_t}}{x}_0+\sqrt{1-\overline{{\alpha }_t}}\epsilon$$

(4)

Through the application of Equation (4), the state of the sample image $x_t$ is obtained after the addition of noise at any time $t$ . Figure 3 illustrates an example of a forward diffusion process applied to a concrete crack image using a linear schedule.

Figure 3.

The Progressive degradation of a concrete crack image using a linear noise schedule.

The reverse diffusion process is the opposite of the forward diffusion process. It is no longer about finding the distribution of $x_t$ under the condition of $x_{t-1}$ , but about finding the distribution of $x_{t-1}$ under the condition of $x_t$ , that is, converting it into a mathematical problem to solve the distribution $q(x_{t-1}|x_t)$ . Assume that $x_0$ is known, that is, the real image is known. The probability distribution under the condition of $x_0$ is expressed as follows:

$$p(x_t|x_{t-1},x_0)=\sqrt{{\alpha }_t}{x}_{t-1}+\sqrt{1-{\alpha }_t}\epsilon$$

(5)

$$p(x_{t-1}|x_0)=\sqrt{\overline{{\alpha }_{t-1}}}{x}_0+\sqrt{1-\overline{{\alpha }_{t-1}}}\epsilon$$

(6)

$$p(x_t|x_0)=\sqrt{\overline{{\alpha }_t}}{x}_0+\sqrt{1-\overline{{\alpha }_t}}\epsilon$$

(7)

Equations (5)–(7) can be rewritten into the Gaussian function form as follows:

$$p(x_t|x_{t-1},x_0)\propto \exp (-\frac{{(x_t-\sqrt{{\alpha }_t}{x}_{t-1})}^2}{2(1-{\alpha }_t)})$$

(8)

$$p(x_{t-1}|x_0)\propto \exp (-\frac{{(x_{t-1}-\sqrt{\overline{{\alpha }_{t-1}}}{x}_0)}^2}{2(1-\overline{{\alpha }_{t-1}})})$$

(9)

$$p(x_t|x_0)\propto \exp (-\frac{{(x_t-\sqrt{\overline{{\alpha }_t}}{x}_0)}^2}{2(1-\overline{{\alpha }_t})})$$

(10)

According to the Bayesian formula, it can be derived that:

$$p(x_{t-1}|x_t)=p(x_t|x_{t-1})\frac{p\left(x_{t-1}\right)}{p\left(x_t\right)}$$

(11)

Using the Bayesian formula in Equation (11), and substituting Equations (8)–(10) into $p(x_t|x_{t-1},x_0)$ , the following expression is obtained:

$$\begin{array}{l}p\left(x_{t-1}|x_t,x_0\right)\propto exp\\ \left(-\frac{1}{2}\left(\frac{{\left(x_t-\sqrt{{\alpha }_t}{x}_{t-1}\right)}^2}{{\beta }_t}+\frac{{\left(x_{t-1}-\sqrt{\overline{{\alpha }_{t-1}}}{x}_0\right)}^2}{1-\overline{{\alpha }_{t-1}}}-\frac{{\left(x_t-\sqrt{\overline{{\alpha }_t}}{x}_0\right)}^2}{1-\overline{{\alpha }_t}}\right)\right)\end{array}$$

(12)

Rearranging Equation (12) into a general form obtains:

$$\begin{array}{c}p\left(x_{t-1}|x_t,x_0\right)\propto exp(-\frac{1}{2}[\left(\frac{{\alpha }_t}{{\beta }_t}+\frac{1}{1-\overline{{\alpha }_{t-1}}}\right){x_{t-1}}^2\hfill \\ -(\frac{2\sqrt{{\alpha }_t}}{{\beta }_t}{x}_t+\frac{2\sqrt{\overline{{\alpha }_{t-1}}}}{1-\overline{{\alpha }_{t-1}}})x_{t-1}+C(x_t,x_0)])\hfill \end{array}$$

(13)

According to Equation (13), the mean and variance can be solved as follows:

$${\sigma }^2=\frac{1-\overline{{\alpha }_{t-1}}}{1-\overline{{\alpha }_t}}{\beta }_t$$

(14)

$$\mu =\frac{\sqrt{{\alpha }_t}(1-\overline{{\alpha }_{t-1}})}{1-\overline{{\alpha }_t}}{x}_t+\frac{\sqrt{\overline{{\alpha }_{t-1}}}{\beta }_t}{1-\overline{{\alpha }_t}}{x}_0$$

(15)

Since it is initially assumed that $x_0$ is known, but in reality $x_0$ is unknown and only $x_t$ is known, an expression for $x_0$ needs to be constructed as follows:

$$x_0=\frac{x_t-\sqrt{1-\overline{{\alpha }_t}}{\epsilon }_t}{\sqrt{\overline{{\alpha }_t}}}$$

(16)

Substituting Equation (16) into Equation (15), the mean expression is obtained as:

$$\mu =\frac{1}{\sqrt{{\alpha }_t}}(x_t-\frac{1-{\alpha }_t}{\sqrt{1-\overline{{\alpha }_t}}}{\epsilon }_t)$$

(17)

Assume that the mean expression obtained by neural network learning is:

$${\mu }_{\theta }=\frac{1}{\sqrt{{\alpha }_t}}(x_t-\frac{1-{\alpha }_t}{\sqrt{1-\overline{{\alpha }_t}}}{\epsilon }_{\theta }(x_t,t))$$

(18)

After obtaining the mean, the mean square error can be used to obtain the loss function:

$$L_t=E_{t,x_0,{\epsilon }_t}[\Vert {\epsilon }_t-{\epsilon }_{\theta }(x_t,t){\Vert }^2]$$

(19)

where $x_t$ is obtained according to Equation (4). According to the loss function, the mean and variance can be trained until they approach the true probability distribution $q(x_{t-1}|x_t)$ .

In the realm of diffusion models, the complexity of noise addition and reversal poses significant computational challenges. To address this, Rectified Flow ³⁷ offers a simplified approach by employing a linear interpolation method for noise addition. Unlike traditional recursive Gaussian models, Rectified Flow introduces noise through a straightforward linear blend between the original image and noise across the diffusion timeline. This method is represented mathematically as:

$$z_t=(1-t)x_0+t\epsilon .$$

(20)

It simplifies both the understanding and computation of noise dynamics. The replacement of complex noise addition equations with Rectified Flow can lead to significant reductions in computational overhead and enhance the model’s transparency.

The training for text-to-image RFM, as guided by the framework in the study by, ³⁸ involves using a pretrained encoder to convert both images and text into latent representations. This approach ensures that both modalities are represented in a compressed format that retains essential information while reducing dimensionality. Images are transformed into a lower-dimensional latent space, which effectively captures the visual content in a more manageable form. Similarly, text conditioning is processed through pretrained text models that convert textual data into corresponding latent representations.

The text conditioning, denoted as $c$ , is encoded using pretrained text models: CLIP-L ³⁹ and CLIP-G. ⁴⁰ The outputs, with dimensions $768\times 1$ and $1280\times 1$ respectively, are concatenated to form a conditioning vector $c_{\mathrm{vec}}$ . The penultimate hidden representations are concatenated channel-wise to form a CLIP context conditioning vector $c_{\mathrm{ctxt}}^{\mathrm{CLIP}}$ . $c$ is further encoded into the final hidden representation $c_{\mathrm{ctxt}}^{T5}$ , derived from the encoder of a T5-v1.1-XXL model. ⁴¹ To align the dimensions with the T5 representation, $c_{\mathrm{ctxt}}^{\mathrm{CLIP}}$ is zero-padded along the channel axis to $4096\times 1$ dimensions. Finally, these representations are concatenated along the sequence axis to produce the ultimate context representation $c_{\mathrm{ctxt}}\in {\mathbb{R}}^{154\times 4096}$ . A pretrained autoencoder ³⁵ is used to compress RGB images into a latent space $x\in {\mathbb{R}}^{h\times w\times 3}$ , where a spatial down sampling factor of 8 is applied. The forward process in the latent space is characterized by rectified flow. The latent representation $x$ is decoded back into pixel space $X=D\left(x\right)$ by a decoder $D(.)$ . ³⁵

The Multimodal Diffusion Transformer ³⁵ architecture, based on diffusion transformer, ⁴² is used for image reconstruction. The Diffusion Transformer integrates the diffusion timestep with the class label, while the Multimodal Diffusion Transformer incorporates embeddings of both the diffusion timestep and text conditioning vectors into its process. Also, a sequence that merges embeddings from both text and image inputs is constructed. Positional encodings are incorporated, and the latent pixel data is transformed by flattening pixel blocks into a sequence of patch encodings. These patch encodings, along with the text sequence, are then adjusted to a common dimensionality and concatenated. Following the Diffusion Transformer, a series of modulated attention mechanisms and Multi-Layer Perceptron (MLPs) are implemented, which are designed separately for the text and image data due to their distinct characteristics. The designs mimic the functions of two transformers, each dedicated to one modality. Subsequently, the sequences from both modalities are combined for the attention mechanism. This allows for an integrated processing where each modality can influence and be influenced by the other.

Prompt generation and prompt engineering

Achieving diversity in generated images requires diverse prompts. Manually writing detailed prompts can be incredibly time-consuming. Fortunately, advancements in the field of natural language processing have enabled LLMs to efficiently generate vast, varied prompts. In this article, GPT-4 is employed to generate text prompts. GPT-4 is a state-of-the-art LLM developed by OpenAI, which includes numerous transformer layers ⁴³ known for their ability to process and generate complex language patterns. Its proficiency in analysing vast amounts of text data enables it to produce varied text prompts, facilitating the creation of unlimited synthesized images.

Not all text prompts generated by GPT-4 can effectively guide the RFM to produce high-quality images. Although these prompts might be well-composed, they do not always resonate with the RFM’s optimal processing capabilities due to a lack of specificity in tuning to the RFM’s preferences. In this study, we use the term ‘RFM-preferred prompt’ to refer to prompts that empirically lead to consistently high-quality image outputs from the RFM. This preference is not derived from fixed linguistic rules or predefined semantic structures. Rather, it reflects the latent tendencies learnt by the RFM during pretraining on large-scale datasets. Since the internal functioning of the RFM is governed by high-dimensional neural representations, the exact preference criteria are not directly interpretable. Instead, we identify these prompts through systematic qualitative evaluation of the images they produce. Figure 4 illustrates an example of RFM-preferred and RFM-non-preferred prompts and the images they generate.

Figure 4.

Comparison of images generated using RFM. (a) shows an image generated from an RFM-preferred, while (b) shows an image generated from an RFM-non-preferred prompt. RFM: rectified flow model

To ensure that RFM can consistently generate diverse, high-quality images that reflect the reality and meet engineers’ preferences, a systematic prompt engineering approach is employed. This process includes the selection of text prompts to enhance their interpretability by the RFM. Figure 5 illustrates the prompt engineering process through which text prompts generated by GPT-4 are selected. Initially, a varied pool of raw prompts is automatically generated using GPT-4. This is done by directly interacting with GPT-4 with specific requests, such as: ‘Hi GPT-4, could you please generate 50 prompts for diffusion models? These prompts should focus on cracks in various civil engineering structures’. GPT-4 then produces a range of text prompts concerning cracks, which include both RFM-preferred and non-preferred types, collectively referred to as raw prompts. Each raw prompt generated by GPT-4 is subsequently used to produce an image using the RFM, with all hyperparameters set uniformly to maintain consistent testing conditions. After the images are generated, a rigorous quality check is performed manually by the proposed prompt engineering strategy. During this evaluation phase, each image is assessed to determine if it meets the user preferences and quality standards. Prompts that lead to the creation of images favoured by users are retained for further use, while those that result in subpar or undesirable outcomes are removed from the pool. Once the quality check is completed, the selected prompts are designated as RFM-preferred prompts and are employed for further image generation.

Figure 5.

The prompt engineering process of selecting RFM-preferred text prompts from a pool generated by GPT-4. RFM: rectified flow model; generative pre-trained transformer (GPT).

To maintain a consistent quality across generated images, each RFM-preferred prompt is used to produce multiple images by randomly varying the RFM’s seed value. The seed value determines the initial noise pattern in the image generation process, setting the starting point for the noise that the RFM manipulates during its diffusion processes. While the RFM is governed by several hyperparameters, our research indicates that the text prompt itself is the paramount hyperparameter, exerting the most significant influence on the quality of the generated images. Modifying a text prompt can dramatically impact the resultant image quality. However, if the prompt is RFM-preferred, subsequent adjustments to other hyperparameters, especially the seed value of the RFM, typically have an insignificant impact on image quality. Although changing the seed may slightly alter image details, it does not significantly affect the overall quality and style of the images. Figure 6 presents an example where the RFM-preferred prompt from Figure 4 is used, maintaining all hyperparameters at their default settings while changing the seed value. It is evident that the image quality remains consistently high, and the style is unchanged but the details, such as the shape and size of the crack, differ.

Figure 6.

The effect of changing the seed value while using the same RFM-preference prompt from Figure 4. RFM: rectified flow model.

Experimental validation of synthesized data for enhanced crack detection

Experiment setup

To evaluate the efficacy of the proposed image generation framework, two distinct experiments involving vision-based tasks, crack classification and crack object detection, are conducted. These experiments are designed to determine how synthesized data, when added to training datasets, can improve the accuracy and robustness of neural networks in detecting and categorizing crack features. The methodologies of evaluation for both the crack classification and crack object detection tasks are depicted in Figure 7. This figure outlines the workflow used for each of the two tasks, from data preparation through model training to the final testing phase.

Figure 7.

Experimental setups and workflow for the crack classification and object detection tasks.

Deep learning-based crack classification task involves training a neural network to classify images as either containing a crack (positive) or not (negative). The primary goal is to assess the network’s ability to accurately identify the presence of cracks based on the visual data provided in the images. Differing from simple classification, the DL-based object detection task requires the trained model not only to detect the presence of a crack but also to pinpoint its exact location within the image. The output includes bounding boxes around each detected crack, accompanied by confidence scores. To perform these evaluations, two public datasets, the Crack Classification Dataset CCLA ⁴⁴ for crack image classification and the Comprehensive Crack Detection Dataset (CCDD)^45–47 for crack object detection are utilized, alongside a synthesized dataset by the proposed framework and the Web-sourced Crack Evaluation Dataset (WCED). The CCLA dataset consists of images of concrete structures from various buildings, with half of the images tagged as negative and the other half as positive. These images are captured using a 16-MP Nikon camera, strategically positioned at a working distance of 500 mm to ensure consistent image quality and scale. Each image within this dataset maintains a resolution of 256 × 256 pixels. The CCDD is compiled by aggregating several public datasets from Roboflow ⁴⁸ for crack object detection. This compilation draws from three datasets,^45–47 summing up to 1020 positive images for training. CCDD and CCLA datasets serve as the base training datasets for the crack classification and crack detection neural networks.

To test the generalization and robustness of the trained models, 400 diverse, real-scenario images from the web are collected and annotated, forming the WCED dataset, which are used only for testing the trained neural networks. WCED dataset comprises 200 negative images of undamaged structures such as buildings and infrastructure, representing scenarios with no crack, and 200 positive images that include a range of crack manifestations, such as small cracks, occluded cracks, cracks obscured by moss and so on. This diverse testing dataset presents a challenging environment for the trained neural networks in detecting and classifying cracks under varied conditions. A few positive images from WCED are displayed in Figure 8.

Figure 8.

Examples of positive images from the WCED dataset. WCED: Web-sourced Crack Evaluation Dataset.

4K synthesized images are generated using the proposed framework as discussed in the second section, 2K positive and 2K negative, to augment the training data. For the crack classification task, these images are integrated with the CCLA dataset. For the object detection domain, 1200 positive synthesized images, also labelled using Roboflow, ⁴⁸ are merged with the CCDD dataset. The integration followed a staged approach, allowing the evaluation of the impact of the synthesized data on model performance. The effectiveness of this strategy is determined by observing any improvements in the models’ performance when testing on the same web-sourced images (WCED), before and after the integration of synthesized data. A noticeable improvement in model performance upon integrating the synthesized images would validate the inclusion of the synthesized dataset and confirm the effectiveness of the proposed synthesized data generation framework.

Synthesized training data generation

For synthesized training data generation, the framework proposed in the second section is used which integrates GPT-4 and RFM. ³⁵ The synthesized crack dataset (SCD) generation begins with the creation of 600 raw prompts related to cracks using GPT-4. From them, 200 RFM-preferred prompts are selected manually through the proposed prompt engineering method introduced in “Prompt generation and prompt engineering” section to optimize prompt suitability for the RFM. Additionally, 600 raw prompts are created for images depicting undamaged structures, from which 200 are chosen following the same prompt engineering process. Upon establishing these 400 RFM-preferred prompts, all hyperparameters are fixed to ensure uniformity across data generation. For each of the selected prompts, 10 unique images are generated by changing the seed value across 10 distinct random numbers. As a result, a total of 2K positive images depicting various crack scenarios and 2K negative images showcasing various undamaged structures are produced. Figure 9 presents three examples produced using the positive prompts. Figure 9(a) displays images generated from a prompt designed for concrete, illustrating a ‘cluster of small, shallow cracks on a concrete wall’. Figure 9(b) shows images from a prompt for asphalt, depicting an ‘asphalt Road surface with a branching crack filled with small vertical pebbles’. Figure 9(c) presents images of a ‘concrete bridge with visible cracks and misalignments from recent earthquake damage, under emergency inspection lights’. These images collectively demonstrate the capability of the proposed synthesized image generation framework to create detailed and context-specific images that reflect different materials and damage scenarios under various conditions.

Figure 9.

Examples of images from the synthesized training dataset generated using RFM with three distinct prompts. (a) Cluster of small, shallow cracks on a concrete wall, (b) asphalt road surface with a branching crack filled with small vertical pebbles, and (c) concrete bridge with visible cracks and misalignments from recent earthquake damage, under emergency inspection lights.

Dataset diversity evaluation

Ensuring dataset diversity is essential for training deep learning models that can generalize effectively to real-world SHM tasks. To evaluate how well different datasets capture variations in crack patterns, we compare the real training dataset (CCDD) and the synthetic dataset (synthetic positive images) against a subset of the real-world testing data (WCED positive images). To quantify the differences between these datasets, we leverage two widely used diversity assessment metrics: Fréchet Inception Distance (FID)^49,50 and Structural Similarity Index Measure (SSIM). ⁵¹ FID measures the similarity between two datasets in the feature space of a pretrained neural network. ⁵² A lower FID score indicates that the dataset is more similar to real-world data. SSIM analyses the structural consistency between images by comparing luminance, contrast, and texture, with a higher score indicating greater similarity to real-world images.

Table 1 presents a quantitative comparison of dataset similarity using FID and SSIM. The results show that CCDD versus WCED (Pos) achieves an FID of 194.9 compared to Synthetic (Pos) versus WCED (Pos) with an FID of 232.5, suggesting that CCDD is more aligned with WCED (Pos) in feature space. However, SSIM, which evaluates structural consistency in the image domain, exhibits the opposite trend. Synthetic (Pos) versus WCED (Pos) achieves a higher SSIM score (0.0427) than CCDD versus WCED (Pos) (0.0244), indicating better local structural similarity to WCED. This apparent contradiction arises from the fundamentally different sensitivities of the two metrics. FID captures overall image structure but is relatively insensitive to fine-grained texture. In contrast, SSIM captures local structural details, such as crack edges and surface patterns, which are critical for SHM. Therefore, while FID suggests greater semantic divergence, the higher SSIM indicates that the synthetic data better reflects the fine-grained textures seen in real crack images.

Table 1.

Diversity metrics comparison.

Metric	CCDD vs. WCED (Pos)	Synthetic (Pos) vs. WCED (Pos)
FID	194.9	232.5
SSIM	0.0244	0.0427

CCDD: Comprehensive Crack Detection Dataset; WCED: Web-sourced Crack Evaluation Dataset; FID: Fréchet Inception Distance; SSIM: Structural Similarity Index Measure.

Moreover, the absolute values of both metrics (e.g., FID ∼195–230, SSIM ∼0.024–0.043) fall outside conventional interpretability ranges. This is likely due to the fact that both metrics are originally designed for tasks requiring near-exact image reproduction (e.g., super-resolution or denoising), whereas our goal is to generate diverse, structurally realistic, but not identical images to real data. Additionally, FID’s reliance on ImageNet-trained networks can underrepresent intra-domain relevance for crack imagery, as these networks are not optimized for civil infrastructure patterns. To mitigate these limitations and provide a more domain-relevant visual interpretation, we present a t-distributed Stochastic Neighbor Embedding (t-SNE) visualization of the image embeddings.

Figure 10 visualizes dataset clustering in feature space using t-SNE. The t-SNE visualization results indicate that Synthetic (Pos) versus WCED (Pos) demonstrates better clustering and alignment compared to CCDD versus WCED (Pos). This aligns with the higher SSIM score observed for Synthetic (Pos), suggesting that it maintains a closer resemblance to WCED (Pos) in terms of structural consistency. Conversely, CCDD versus WCED (Pos) shows a more dispersed distribution in the t-SNE space, which is consistent with its lower SSIM score. These results suggest that, despite its higher FID score, Synthetic (Pos) outperforms CCDD in both SSIM and t-SNE clustering, indicating a stronger overall alignment with WCED (Pos).

Figure 10.

t-SNE visualization of feature distributions, comparing CCDD and Synthetic (Pos) against WCED (Pos). CCDD: Comprehensive Crack Detection Dataset; WCED: Web-sourced Crack Evaluation Dataset.

While these metrics provide valuable insights, they have inherent limitations in capturing dataset diversity, particularly given the limited size of the real testing dataset. To further evaluate the effectiveness of the synthesized dataset, we assess its role in improving crack classification and detection performance.

Synthesized data evaluation – crack classification

The model employed for the crack classification task is a VGG-16 architecture, ¹⁴ initially trained on the ImageNet dataset. ⁵³ The architecture of the neural network is shown in Figure 11. The base model consists of five convolutional blocks, each followed by max pooling layers, effectively capturing hierarchical features from the input images. The first two blocks contain two convolutional layers with 64 and 128 filters, respectively, while the subsequent blocks have three convolutional layers each, with 256, 512 and 512 filters. These layers are activated by the Rectified Linear Unit (ReLU) function and progressively reduce the spatial dimensions of the input through max pooling. The flattened output from the last max pooling layer, a one-dimensional vector of 25088 elements, is fed into two fully connected layers, each with 4096 units and the ReLU activation. Finally, a dense layer with a single unit and the sigmoid activation function is added to output a probability value for binary classification.

Figure 11.

The architecture of VGG-16 neural network. VGG: Visual Geometry Group,

The CCLA dataset is initially employed to train the neural network, establishing a baseline performance. Subsequently, the synthesized dataset from the proposed framework, comprising both positive and negative images, is progressively integrated with the CCLA dataset. This integration follows a staged approach: starting with an addition of 500 positive and 500 negative images, followed by increments to 1K positive and negative, then 1.5K, and finally culminating with 2K of each. The progressive integration aims to quantitatively evaluate the effect of augmentation with purposely-generated synthesized images. All model evaluation is conducted on the WCED dataset. The performance of the models is assessed using three metrics: Precision, Recall and the F1-score. Precision measures the accuracy of the positive predictions for the cracked class:

$$\mathrm{Precision}=\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FP}},$$

(21)

where True Positives (TP) represents the number of correct identifications of cracks, and False Positives (FP) denotes the model incorrectly identified cracks in non-cracked images or locations. Recall is used to evaluate the model’s effectiveness in identifying all actual instances of cracks:

$$\mathrm{Recall}=\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FN}}.$$

(22)

where False Negatives (FN) are cases where the model fails to identify actual cracks. The F1-score combines both Precision and Recall into a single metric by calculating their harmonic mean, providing a balanced measure of the model’s accuracy and completeness:

$$F_1=2\times \frac{\mathrm{Precision}\times \mathrm{Recall}}{\mathrm{Precision}+\mathrm{Recall}}.$$

(23)

A summary of the model performances is given in Table 2. Across different stages of synthesized data augmentation, a significant improvement in performance is yielded. The baseline model, trained solely on the CCLA dataset, achieves an F1-score of 0.347, indicating modest performance that may stem from a limited diversity of training examples, potentially hindering the model’s ability to handle complex crack patterns. The introduction of 500 positive and 500 negative synthesized images (denoted as SCD(P&N)) markedly enhances the model’s capabilities, as evidenced by the F1-score rising to 0.740. This substantial increase suggests that even a moderate addition of the synthesized images from the proposed framework to the training images can significantly boost the model’s ability to generalize across different scenarios. As more synthesized data are progressively added, the F1-scores continue to climb to 0.897 and 0.921, respectively. This trend underscores the enhanced Precision and Recall, illustrating the model’s improving proficiency in accurately identifying and classifying crack images. The optimal performance is observed when the dataset is expanded to include 2K positive and 2K negative synthesized images, culminating in an F1-score of 0.932. The clear benefits in Precision, Recall and overall F1-score through the synthesized data integration affirm the effectiveness of the proposed image generation framework in training better neural network models for crack image classification task.

Table 2.

Performance of VGG-16 across different stages of synthesized data integration.

Training data	Testing data	Precision	Recall	F1-score
CCLA	WCED (P&N)	0.3360	0.3587	0.347
CCLA + 500 SCD(P&N)	WCED (P&N)	0.7596	0.7215	0.740
CCLA + 1K SCD(P&N)	WCED (P&N)	0.8996	0.8952	0.897
CCLA + 1K SCD(P&N)	WCED (P&N)	0.9215	0.9213	0.921
CCLA + 2K SCD(P&N)	WCED (P&N)	0.9396	0.9242	0.932

VGG: Visual Geometry Group; SCD: synthesized crack dataset; WCED: Web-sourced Crack Evaluation Dataset.

Synthesized data evaluation – crack detection

To further evaluate the efficiency of the proposed dataset generation framework, the synthesized dataset is employed to train a crack object detection model. Different from the crack image classification, object detection not only identifies the presence of cracks but also predicts bounding boxes around the cracks, making it a more complex challenge. For this purpose, the YOLOv5 ⁵⁴ neural network is utilized. YOLOv5 is a popular object detection model renowned for its speed and accuracy. The architecture of YOLOv5 is based on the Darknet model, ⁵⁵ but it has been optimized and improved to enhance performance. It features a backbone designed for feature extraction, a neck responsible for feature integration, and a head for bounding box prediction. The backbone is based on the CSPDarknet53, ⁵⁶ which reduces the computational complexity and improves the learning capability of the model through cross-stage partial connections that allow for more efficient feature propagation. In the neck, YOLOv5 uses a series of feature pyramid networks ⁵⁷ and path aggregation networks ⁵⁸ to merge features from different stages of the backbone, enhancing the model’s ability to detect objects at various scales.

Similar to the crack classification experiment, the object detection experiment begins by utilizing the CCDD to fine-tune a YOLO, establishing a baseline crack detection model. Subsequently, 200 positive images from the WCED are annotated and used to conduct an initial evaluation of the model’s performance, providing base metrics for comparison. Following this initial assessment, synthesized images are progressively integrated into the training dataset in stages: 200, 400, 600, 800 and up to 1.2K images. To evaluate the accuracy of the neural network for crack detection, four key metrics are utilised: Precision, Recall, mAP-0.5 and mAP-0.5:0.95. The metrics mAP-0.5 and mAP-0.5:0.95 are for evaluating the precision and reliability of object detection models, particularly in their ability to accurately predict bounding boxes around detected objects. mAP-0.5 measures how closely the model’s predicted bounding boxes align with the ground truth boxes, requiring at least a 50% overlap. It calculates the average Precision (AP) for each class at this threshold, where precision is the proportion of correct positive predictions and recall is the detection of actual positives. The mAP-0.5:0.95 offers a more stringent assessment by averaging APs across a range of Intersection over Union (IoU) thresholds, from 0.5 to 0.95 in increments of 0.05. This metric evaluates the model’s accuracy at various levels of strictness in bounding box overlap, from the minimum acceptable overlap to near-perfect precision.

Table 3 and Figure 12 depict the performance of a YOLOv5 neural network fine-tuned on an incrementally enlarged SCD dataset. Each training configuration within the table is differentiated by the number of synthesized images added to the initial CCDD dataset. All trained models are consistently tested on positive images from the WCED dataset. The consistent improvement in Precision and Recall metrics with the addition of synthesized data indicates that the extra images significantly enhance the model’s ability to accurately identify and classify crack features. The mAP at various IoU thresholds provides further insights into the model’s accuracy. Both mAP metrics show marked improvements as more synthesized data is integrated, highlighting a better alignment of predicted bounding boxes with actual crack positions. The progression of model performance metrics, from the baseline using only the CCDD to subsequent stages with added synthesized images, shows significant enhancements. Notable performance gains are observed with the first additions of synthesized data (200 and 400 images), though the rate of improvement slows slightly as more data is added. Figure 13 showcases some examples of crack detection performance on the WCED dataset. In these images, a label of ‘0’ indicates a detected crack, followed by a confidence value. For all analyses, a confidence threshold of 0.5 is established; only detections exceeding this threshold are considered valid. The depicted cracks are initially undetectable by the YOLOv5 model when solely fine-tuned on the CCDD dataset. Subsequent fine-tuning of YOLOv5 with the addition of 1200 positive images from the SCD dataset markedly enhanced the model’s detection capabilities.

Table 3.

Performance of YOLOv5 across different stages of synthesized data integration.

Training data	Testing data	Precision	Recall	mAP-0.5	mAP-0.5:0.95
CCDD	WCED(P)	0.2425	0.3818	0.2983	0.0762
CCDD + 200 SCD(P)	WCED(P)	0.5053	0.5636	0.5075	0.2632
CCDD + 400 SCD(P)	WCED(P)	0.6700	0.6873	0.6599	0.4346
CCDD + 600 SCD(P)	WCED(P)	0.7185	0.7818	0.7429	0.5252
CCDD + 800 SCD(P)	WCED(P)	0.7656	0.8009	0.7946	0.6325
CCDD + 1K SCD(P)	WCED(P)	0.8305	0.8272	0.8118	0.6644
CCDD + 1.2K SCD(P)	WCED(P)	0.8382	0.8454	0.8374	0.6797

CCDD: Comprehensive Crack Detection Dataset; WCED: Web-sourced Crack Evaluation Dataset; SCD: synthesized crack dataset.

Figure 12.

Line graph showing the progression of Precision, Recall and mAP at different stages of synthesized data integration in YOLOv5 training. mAP: mean Average Precision.

Figure 13.

Examples of enhanced crack detection performance on the WCED dataset using YOLOv5, post-integration of the entire SCD dataset for fine-tuning. WCED: Web-sourced Crack Evaluation Dataset; SCD: synthesized crack dataset.

Conclusion and future work

In this study, a novel framework for synthesized image dataset generation is proposed to enhance the capabilities of DNN models in structural crack detection. This framework combines the RFM with GPT-4, along with a prompt selection process to generate an ideally unlimited number of synthesized crack images. It enables the simulation of a broad spectrum of crack scenarios (from minor surface cracks to major structural damages) that can be inaccessible under normal operational conditions. Additionally, the image generation process is highly efficient, producing thousands of images in just a few hours hence significantly speeding up the training and development cycles for SHM applications. Experiments demonstrate significant improvements in crack image classification and object detection performance metrics. These enhancements validate the usage of the synthesized datasets in addressing the challenges associated with the limited and homogeneous training data typically available in SHM. Despite the high efficiency of image generation, manual annotation of the synthesized images remains essential. Future advancements in automating the annotation process, such as for crack segmentation tasks, could significantly streamline the workflow. This would help reduce both time and labour costs, ultimately enhancing the overall effectiveness and scalability of SHM systems.