Abstract
Bridges are often located in harsh environments and are thus extremely susceptible to damage. If the initial damage is not detected in time, it can develop further causing safety hazards. Therefore, accurate detection of bridge damage is an important topic. In recent years, artificial intelligence technology has been developed rapidly, especially machine learning algorithms, which have shown amazing results in various fields while it also received attention in bridge inspection. This paper summarizes the progress of bridge damage detection research related to artificial intelligence techniques between 2015 and 2021. For structural health monitoring, sensing data is the basis for various data processing methods. The strength and weakness of the sensing data itself directly affect the effectiveness of subsequent processing methods. As a result, this paper classifies bridge damage detection studies into six categories from the types of sensing data: visual image, point cloud, infrared thermal imaging, ground-penetrating radar, vibration response, and other types of data. These six types of damage detection methods were reviewed and summarized respectively. Finally, challenges and future trends were discussed.
Introduction
As one of the important infrastructures, bridges significantly contribute to the economic development of a country. By the end of 2019, the Chinese road bridge stock had exceeded 870,000. 1 The Hong Kong-Zhuhai-Macao Bridge, which was opened to traffic in 2018, is the world’s longest bridge-tunnel combination highway across the sea, with a total cost of approximately RMB127 billion. 2 Most of the places where bridges need to be built have harsh service environment, while the bridges are subject to damage caused by material degradation and external loads, etc. Modern bridge construction in the United States was earlier. After a century of construction and development, the number of bridges in the United States increased from 1800 in 1900 to 474,700 in 2015. Among them, small and medium span bridges less than 50 m account for 97.8% of the total number of bridges in the US. 3 Because of the early construction of bridges in the US, much of which began in the 1930s, a large number of old bridges appeared earlier. In order to manage bridge safety comprehensively, the Federal Highway Administration has compiled the National Bridge Inventory (NBI). 4 It includes identification information, bridge type, operating conditions, bridge geometry data, inspection data, etc. The NBI has grouped unsafe bridges into two categories: structural defects and functional failures. Similarly, in 2013, the Japanese government established the “Infrastructure Maintenance, Renovation and Management” in response to the aging of public facilities such as bridges and embankments. 5 This plan specifically points to the use of sensors, robots and non-destructive inspection techniques, etc., to improve the level of inspection and repair. According to statistics on bridge accidents in Europe and the US, most bridge collapses occur between 50 and 100 years of service. 6 At present, most bridges are gradually reaching the end of their design life, and structural deterioration and material deterioration are gradually increasing. It is, therefore, important to monitor the safety of bridge structures. Bridge safety involves all aspects and damage identification is one of the main tasks in safety inspection.
Bridge damage detection relies on advanced sensing techniques to obtain the dynamic response, external loads and the apparent morphology of the bridge structure. Structural Health Monitoring (SHM) focuses on strategies and processes for damage identification and characterization of engineering structures. 7 In order to improve the monitoring of engineering structures, more and more structural health monitoring systems are being installed on bridge structures. As of 2016, more than 80 bridges in Hong Kong and Chinese mainland alone were equipped with SHM systems. 8 A health monitoring system for a large span bridge consisting of more than 50–300 sensor measurement points. These structural health monitoring systems continuously collect various types of sensing data from bridges and provide data support for artificial intelligence-based bridge damage identification. In recent years, with the rapid development of artificial intelligence, it has been widely used in various engineering fields, such as intelligent transportation,9–11 smart construction sites,12–14 and structural damage monitoring.14–17 Artificial intelligence (AI) is a branch of computer science that aims to simulate human intelligence with machines. AI systems use techniques such as machine learning, particularly deep learning, to present “intelligent” behavior. 18 Machine learning algorithms can be divided into four categories according to the learning method: supervised learning, unsupervised learning, semi-supervised learning, and reinforcement learning. 19 Supervised learning uses samples with known labels to train the network; Unsupervised learning uses unlabeled samples to train the network; Semi-supervised learning uses a large number of unlabeled samples and a small number of samples with known labels to train the network; and Reinforcement learning uses unlabeled sample data and a rewards function to train the network. Because the external environment provides little information, reinforcement learning requires experience and experimentation to learn on its own. Since supervised learning exhibits high accuracy rates, a large number of relevant research papers have focused on supervised learning. 20 According to the learning task, machine learning can be divided into classification algorithms, regression algorithms and clustering algorithms. Classification algorithms predict the category labels of data samples, and the predictions are discrete for example, damage classification. 21 The regression algorithm predicts a continuous value for a sample of data, such as the prediction of the compressive strength of concrete. 22 Clustering algorithms are used to predict the class labels of data samples by unsupervised learning methods. Bridge damage identification uses a large amount of sensor data to determine the location and extent of bridge damage. These three types of problems can therefore be well integrated with the bridge damage detection task.
The purpose of this review is to survey the latest research in artificial intelligence for bridge damage detection, with the aim of identifying state-of-the-art examples that facilitate bridge safety inspection and assessment research. Several advanced review papers have been published in the field of bridge damage detection. Wang et al. 23 summarized the research progress of different Non-Destructive Testing (NDT) and assessment methods in bridge damage. Arangio 24 provided a summary of the neural network-based approach to bridge damage identification and revisited the neural network model from a Bayesian probabilistic logic perspective. Malekjafarian et al. 25 and Yang and Yang 26 reviewed indirect bridge detection based on passing vehicles, respectively. Moughty and Casas 27 conducted a technical review of modal-based bridge damage detection. Casas and Moughty 28 reviewed damage detection of small and medium span bridges based on vibration data. An et al. 29 reviewed and presented bridge damage identification based on different data processing methods. However, AI algorithms have been widely developed and adopted in 2015 and especially after 2017. Sun et al. 30 reviewed big data and artificial intelligence-assisted bridge condition assessment and damage detection. They provided an overview of three aspects of structural health monitoring, big data and AI. However, we focus on the latest research on the organic integration of different types of sensing techniques with AI in the field of bridge damage detection.
This paper is concerned with the application of AI to bridge damage detection, reviewing bridge damage detection methods according to the type of sensed data. Different sensing technologies have different emphases, resulting in different types of sensing data being used to address different bridge damage. Machine learning is a class of data-driven algorithms designed to fully exploit the high-dimensional spatial features in sensed data for damage detection. Therefore, sensed data is the basic and most important core of all types of detection methods. In Section 2, due to space constraints, some representative works between 2015 and 2021 are summarized. In particular, Section 2.1 provides an overview of the image-oriented methods. Section 2.2 shows the point cloud-oriented methods. In contrast to images and point clouds, which focus only on surface damage, infrared thermography can enhance both the identification of surface damages and the detection of internal damages closer to the surface of components. Therefore, Section 2.3 presents the infrared thermography-oriented methods. To detect internal damage to bridges at greater depths, ground-penetrating radar (GPR) is often used. Section 2.4 summarizes the GPR-oriented methods. However, not all bridge damages have morphological characteristics. For more subtle damage such as stiffness attenuation, it is necessary to rely on dynamic properties for identification. Section 2.5 summarized the vibration response-oriented methods. Section 2.6 demonstrates other types of data-oriented methods. Section 3 discusses the challenges of current technology and future developments. Section 4 concludes this paper.
Artificial intelligence solutions for bridge damage detection
Data, the core of AI algorithms, is a key factor in determining how well AI can learn to be “intelligent.” With the development of various sensing technologies and SHMs, various types of sensed data are being generated. In current SHMs, the most widely used data is vibration response. However, with the rapid development of deep learning in the field of image processing, damage detection methods based on image and 3D point cloud data have been substantially studied in recent years. Images and 3D point clouds are obtained based on optical sensing techniques. These two types of sensed data cannot by themselves characterize non-apparent damage. To extend the limits of these two sensing technologies, sensing technologies such as infrared thermography and ground-penetrating radar are also gradually being combined with machine learning algorithms. Therefore, this section provides an overview of AI-based bridge damage detection methods in six categories: image, point cloud, infrared thermography (IRT), ground-penetrating radar (GPR), vibration response, and other types of data, as shown in Figure 1.
Overall framework of this paper.
Image-oriented methods
Visual inspection is a common inspection method used in structural health monitoring. However, visual inspection relies heavily on the inspector’s own judgment. In addition, the visual inspection of some elements is very difficult and dangerous for bridge structures. Damage detection under bridges often relies on bridge inspection vehicles, a task that is extremely costly and dangerous. Visual means such as unmanned aerial vehicles (UAVs) and cameras can be a good alternative to visual inspection of bridge damage. Although UAVs can easily photograph and record the exterior of each bridge element, this process generates a large amount of data. To further reduce human involvement, image-oriented intelligent damage identification methods have received extensive attention and research. Meanwhile, machine learning algorithms such as deep learning have achieved significant results in the image processing. As a result, vision and machine learning-based methods for bridge damage identification have been well researched. These methods are mostly applied to bridge surface damage identification. This section is divided into three categories according to the surface damage detection task: surface damage classification, surface damage identification, and surface damage segmentation.
Surface damage classification
In bridge surface damage classification, captured images are divided into image blocks with the same size. These image blocks are then classified by machine learning algorithms such as deep learning as to whether they contain bridge damage or not. Therefore, machine learning methods are used to address a classification task in this section. Machine learning methods such as Artificial Neural Network (ANN), Support Vector Machine (SVM), and Extreme Learning Machine (ELM) can all be used for classification tasks. However, in the field of image processing, local features are very important for image target detection. Since most traditional machine learning algorithms cannot accurately capture local features in images, deep convolutional neural networks are widely used. In a deep convolutional neural network, the convolutional kernel slides over the image to extract local features. As the convolutional layers are deepened, the extracted features gradually become abstract, which in turn completes the classification of the image. 31 In classification tasks, some excellent convolutional neural networks have achieved high precision results. Table 1 shows some classical convolutional neural networks for classification. It is worth noting that these convolutional neural networks can also be used for regression problems by simply modifying the last layer and the loss function. Many bridge surface damage identification methods used these classical networks to classify damage. The effectiveness of these classical networks has been demonstrated in large datasets. These large datasets contain a large sample of data, such as ImageNet, 32 which was created as a computer vision dataset led by Professor Fei-fei Li at Stanford University and contains over 14 million images. For the engineering field, it is difficult for us to obtain such huge amounts of monitoring data.
Some classical convolutional neural networks for classification.
These classical networks contain a large number of parameters that need to be trained and can easily be overfitted or have difficulty in achieving high identification accuracy on small datasets. Pre-trained models based on large datasets and transfer learning can be a good solution to this problem. This is the main reason why most researchers use these classical networks. Dung et al. 39 compared the effectiveness of shallow CNN and VGG16 in the early detection of fatigue cracks in steel bridge. They captured 167 RGB images of 1024 ×768 pixels which were divided into 32,064 small images of 64 ×64 pixels. These small images were labeled “cracked” or “non-cracked” for training purposes. The test results showed that the fine-tuning pre-trained VGG16 achieves the highest classification performance. As most of the images in the dataset were marked as “non-cracked,” the images marked as “cracked” were augmented in order to reduce the category imbalance. They used data augmentation methods including rotation, offset, flip, fill, clip, and zoom. Data augmentation could improve classification accuracy by up to 5%. Cha et al. 40 proposed a shallow CNN method for surface crack classification. The experimental results showed that the proposed classifier achieved 98% of the testing accuracy. Their results are significant compared to traditional edge detection methods such as Canny and Sobel. For better compatibility with transfer learning, Chen 41 changed the configuration to a fully connected layer with 256 neurons before the Softmax layer in the VGG network. Besides, a batch normalization (BN) layer was added between the convolution and pooling layers, which can effectively suppress the occurrence of gradient dispersion. Their results showed that transfer learning could reduce the training time of the network while it could avoid overfitting.
It has been shown in many studies that transfer learning based on pre-trained deep learning models can achieve good results in classifying images with or without damage. However, in each particular task, there is a trade-off between the number of parameters that need to be trained and the number of parameters from transfer learning that do not need to be trained. Therefore, the optimization of transfer learning needs further research. Aliyari et al. 42 used multiple classical deep convolutional neural networks to classify bridge crack damage. The more parameters for training, the more time it takes to train. Their study showed that for ResNet50, an increase in the number of parameters to be trained significantly reduces the accuracy of classification. The optimal number of parameters to be trained for ResNet50 should be between 17,540,706 and 24,124,770. In the case of VGG16, transfer learning cannot significantly improve its classification accuracy. In order to improve the effectiveness of convolutional neural networks, some new forms of convolution have been proposed. For example, atrous convolution allows for a larger receptive field without reducing resolution. Depthwise separable convolution significantly reduces the computational complexity. Xu et al. 43 proposed a new convolutional neural network architecture consisting of depthwise separable convolution and Atrous Spatial Pyramid Pooling (ASPP). This is a new network structure, so there is no corresponding pre-trained model to complete the transfer learning. Even so, the proposed method achieved 96.37% accuracy in classifying surface cracks on bridges. Therefore, it is also necessary to develop new types of networks based on classical structures. Although various types of bridge surface damage classification methods have been proposed, there is a lack of the unified large datasets to evaluate these methods. Both the number and diversity of samples in the dataset used differ from the complex environment in actual engineering inspections. There is still no reliable assessment on the effectiveness of their tests during the actual testing process. Furthermore, although classification networks achieve a relatively simple task, classification networks are the basis for recognition and segmentation networks. The further development of highly efficient and highly accurate classification networks therefore continues to be of great importance.
Surface damage identification
Bridge surface damage identification is the use of machine learning algorithms to identify and locate bridge damage in images. Compared to the damage classification task, damage identification not only determines which type of bridge damage is contained in images, but also requires autonomous localization of the damaged part of the bridge using the identification box. Thus, the damage identification essentially consists of a classification task and a regression task. Classification algorithms such as VGG16 can also be combined with traditional sliding window technique to identify and localize bridge damage in images. However, this damage identification method is very time consuming, and the locating box cannot be adjusted autonomously according to the size of bridge damages. For this reason, the bridge damage identification method has been further developed on the basis of the bridge damage classification method. Some classical object detection networks are shown in Table 2. The feature extraction part from these object detection networks is mostly inspired by the architecture of classification networks. These object detection networks have also evolved. For example, the You only look once (YOLO) has been developed from Version 1 (V1) to Version 5 (V5). Only the original versions of these networks are listed in Table 2. These object detection networks can be divided into one-stage and two-stage methods according to their detection characteristics. The two-stage method first seeks out some candidate regions before adjusting and classifying them, whereas the one-stage method only needs to extract features once to achieve object detection. As a result, the one-stage method is more efficient.
Some classical object detection networks.
To facilitate bridge damage detection, drones combined with intelligent algorithms can replace personnel for rapid inspection of bridge high-risk components such as piers and abutments. Kim et al. 49 used UAVs and R-CNN to locate and identify cracks on bridge surfaces. They collected 384 crack images with 256 × 256 pixel as the dataset. Then, two types of crack parameters (area and width) were quantified according to the extracted crack image blocks. Ma et al. 50 presented a UAV-assisted machine learning tool for bridge damage detection after earthquakes. They used Faster R-CNN to quickly identify and locate cracks and spalling on bridge surfaces. Experimental results showed that the classification accuracy of the method is 84.7%. This method was also used to detect damage on the Anzac Bridge in Christchurch. In order to improve the identification of existing network structures, some optimization theories are used. Song et al. 51 proposed a preliminary selection and refined identification scheme based on the Faster R-CNN. This optimized solution was used to identify three types of bridge surface damage: honeycomb pitting, crack, and salt out. Compared with Faster R-CNN, this method improved the mean Average Precision (mAP) of the three categories by 23.89%, 21.04%, and 35.43% respectively; its omission rate was reduced by 10.81%, 9.94%, and 11.27% respectively. Yu et al. 52 used a k-mean clustering algorithm to obtain anchor sizes that corresponded to the size of damages. In order to be able to visually characterize the true size of the damage, the shooting distance and focal length are fixed during the data acquisition phase. The proposed method was used for the identification of three common types of damage on bridge surfaces: cracks, spalling and exposed reinforcement. The identification results showed that the average identification accuracy for the three types of damage is 84.56%, which is higher than that of the Faster R-CNN with predefined anchor points.
In contrast to parametric optimization, some new modular structures and convolutional forms can be used to facilitate the development of object detection networks. Yang et al. 53 improved the YOLOV3 using the Spatial Pyramid Pooling (SPP) module. Meanwhile, transfer learning strategy based on the pre-trained model was used to learn a small sample of concrete bridge damage data. The experimental results showed that the SPP module could improve mAP by 1.3%; transfer learning could improve mAP by 20.9%. Chen et al. 54 used deformable convolution to extract more accurate features based on YOLOV3. Similarly, transfer learning was also used to improve the identification accuracy of rust, collapse, cracks and weeds on the bridge surface. They analyzed the effect of the number of transfer layers on the identification results. The highest mAP of 84% was obtained when the number of transfer layers was 10. Moreover, pruning and group convolution were used to compress the model in order to speed up the inference speed of models. When the compression rate is 0.1, the mAP is 84.4%; When the compression rate is 0.3, the mAP is 75.8%. The higher the compression rate, the worse the identification effect of the compressed model.
Surface damage segmentation
The bridge damage classification method and the bridge damage identification method can provide technical support for the rapid inspection of bridge surface damage. However, these two types of bridge damage detection methods do not provide further refinement of information. Therefore, in order to be able to obtain quantitative parameters for bridge surface damage, bridge damage segmentation methods based on pixel segmentation are proposed. Pixel segmentation can be divided into semantic segmentation, 55 instance segmentation, 56 and panoramic segmentation. 57 Here, the differences between these three segmentation methods are briefly described. Semantic segmentation and instance segmentation perform pixel-level identification of only the target objects in an image, whereas panoramic segmentation performs pixel-level identification of all objects that appear in an image. Furthermore, in contrast to semantic segmentation, instance segmentation can assign different IDs to multiple targets in an image. As an example, suppose an image has two bridge surface cracks. Both semantic segmentation and instance segmentation allow for pixel-level identification of the two cracks. Instance segmentation can also assign different IDs to the two cracks, while semantic segmentation cannot distinguish between the two cracks. Nevertheless, the differentiation of multiple damages is of little significance for bridge damage identification. Therefore, semantic segmentation methods are heavily used in bridge damage identification. Some classic segmentation algorithms are shown in Table 3. During the development of semantic segmentation algorithms, new forms of connectivity and convolution have been adopted.
Some classic segmentation algorithms.
The damage classification method only classifies image blocks, and the damage identification method classifies and regresses rectangular candidate boxes of different sizes in an image. However, pixel-level segmentation requires the classification of each pixel point in an image. As a result, pixel-level segmentation network is computationally very intensive and difficult to train compared to damage classification and damage identification. Rubio et al. 63 proposed an FCN-based pixel-level identification method for concrete bridge delamination and reinforcement exposure. In this method, FCN used the pre-trained VGG as the feature extractor. Experimental results based on the bridge data from Niigata showed an average accuracy of 89.7% for delamination and 78.4% for reinforcement exposure. Shi et al. 64 used U-Net to segment steel corrosion and rubber bearing cracks in bridges. Similarly, the encoding structure in U-Net used the pre-trained VGG. In addition, the effect of two different image input methods on the segmentation results was analyzed. The first input method is to compress the high-resolution image directly and input it into the network, the second input method is to crop the high-resolution image to 224 × 224 pixel and then input it into the network. The results showed that for large damage such as corrosion, the second method achieves better segmentation, while for small damage such as cracks, the first method achieves better segmentation. Scour is also a very important type of bridge damage.65,66 Lin et al. 67 used Mask R-CNN and images to obtain the flood water level changes around the bridge piers. The image identification results were used to validate the simulated water levels obtained from the hydrodynamic model, thus improving the evolution accuracy of the bridge scour depth.
In order to improve the identification accuracy of semantic segmentation, several new modules have been used to improve the classical network structures. Deng et al. 68 added an Atrous Spatial Pyramid Pooling (ASPP) module to LinkNet and used the pre-trained ResNet-34 as the encoder to avoid overfitting small datasets. The proposed method was used to identify bridge delamination and reinforcement exposure at pixel level. The number of pixels taken up by damage in an image is much smaller than the number of pixels taken up by the background. Thus, weight-balanced Intersection of Unions (IoU) loss functions were used to achieve accurate segmentation on small data sets that are highly unbalanced. Their experimental results showed that the proposed method achieves better segmentation than U-Net and LinkNet. Fu et al. 69 introduced Dense ASPP in the DeepLabv3 for segmenting bridge cracks. The proposed method achieved an mIoU of 82.37%, which is better than that of the original DeepLabv3. Li et al. 70 added a BN layer and an ASPP module to the FCN for surface crack segmentation of full-size steel bridges in complex backgrounds. Similarly, the pre-trained ResNet50 was used in the encoding structure. Finally, the proposed method achieves an mIoU of 74.08% in a complex background, outperforming benchmark algorithms such as LinkNet, DeepLabV3 and SegNet.
It requires large amounts of annotated data for bridge damage classification, identification and segmentation tasks. Especially for the segmentation task, which requires manual annotation of every pixel in the image, this workload is enormous. Karaaslan et al. 71 proposed an attention-guided semi-supervised bridge damage detection method. The SSD was first used to acquire regions of interest for bridge surface damage, and the proposed regions were fed into an attention guided SegNet to complete the damage segmentation. The experimental results showed that the proposed method significantly reduces the computational cost and the number of samples to be trained. Most pixel segmentation algorithms contain a large number of parameters, making model inference inefficient and unsuitable for mobile devices. In order to balance detection efficiency and accuracy, more concise and efficient damage segmentation networks have been proposed. Lopez Droguett et al. 72 proposed a DenseNet architecture with only 13 layers for pixel-level segmentation of bridge surface cracks. The proposed architecture has 350,000 parameters to achieve an IoU of 94.51%. Li et al. 73 combined Short-Term Dense Concatenate network and refinement network for bridge surface crack segmentation. The former improves detection efficiency by eliminating redundant structures, while the latter combines shallow spatial details with deep semantic information to improve detection accuracy. The validation results using the three datasets showed that the proposed method achieves a detection accuracy of 97.54% and a detection efficiency of 37 images per second.
For deep learning networks, a large number of hyperparameters need to be specified by humans, and these parameters can affect the identification effectiveness of the model. In order to reduce human involvement while improving model identification accuracy, Bayesian optimization was used to select deep learning hyperparameters to obtain a more robust bridge damage identification model. Liang 74 completed system-level fault classification, component-level bridge pillar detection and local damage level localization at three levels: image classification, target detection, and semantic segmentation, respectively.
Point cloud data-oriented methods
The images only reflect two-dimensional (2D) information about the bridge damage. In order to obtain three-dimensional (3D) information about the bridge damage, point cloud data is acquired. Point cloud data is a set of vectors in a three-dimensional coordinate system. Points contain a rich variety of information, such as 3D coordinates, colors, intensity values, etc. 75 Point clouds can be obtained by a 3D laser scanner or by 3D reconstruction from 2D images. In addition, the depth map acquired by the depth camera can be converted to obtain the point cloud. In contrast to two-dimensional images, each pixel in a depth map has four attributes, namely red (R), green (G), blue (B), and depth (D). 76 A comparison of the images with the point cloud data is shown in Table 4. In contrast to point cloud data, image data is ordered. In the case of a gray-scale image, it is essentially a matrix where each value represents the gray value of that pixel. The color image is created by superimposing three matrices with the same size. The values in the three RGB matrices represent the corresponding color values of the pixel. In an image, each pixel is connected sequentially, so the image is an ordered data structure. However, in the most basic point cloud data, each point is composed of three coordinate values. There is no strict order between the individual points.
Comparison of images and point clouds.
The 3D point cloud information of the bridge is obtained to complete the 3D modeling of the bridge during its operation and maintenance cycle. Geometric quality inspection and bridge maintenance management can be completed based on the modeling results at different stages. 77 In order to segment the target object from the point cloud data, some classical point cloud processing networks have been proposed, as shown in Table 5.
Some classic point cloud processing networks.
Nasrollahi et al. 83 proposed a PointNet-based method for the 3D identification of bridge surface defects. They collected five-point cloud datasets from five areas of the concrete bridge, which included 3572 labeled samples. These samples consist of two categories: defect and non-defect. PointNet could achieved a defect recognition accuracy of 65.9%. Kim et al. 84 proposed a deep learning-based method for automatic identification of bridge components. The collected point cloud data was first divided into different subspaces. These subspaces were fed into PointNet to complete the semantic segmentation. Finally, consensus labels were used to identify bridge components. The point cloud data from seven full-size bridges were tested and showed that the point cloud segmentation could be effectively improved by increasing the overlap between subspaces. Then, the effectiveness of PointNet, PointCNN and DGCNN in bridge component identification was compared and studied. The experimental results showed that the mIoU of DGCNN is 86.85%, which is higher than that of PointNet (84.29%) and PointCNN (79.68%). In order to further improve the identification of DGCNN. In order to improve the effectiveness of these classical networks in 3D point cloud processing, some improved solutions have been analyzed and adopted. Lee et al. 85 introduced ring-type neighbor to extract a wider range of neighbors on top of DGCNN. The 17 bridge decks of the four bridges were scanned using a ground-based laser scanner, while the point cloud data was RGB mapped with additional camera-captured camera measurements. Therefore, these point cloud data include 3D coordinates and RGB information. The experimental results show an improvement in the overall accuracy of the proposed method for segmenting bridge members from point cloud data, with an improvement of about 3% in mIoU. Xia et al. 86 proposed an automatic semantic segmentation method for bridge point clouds based on local descriptors and machine learning. It selects a custom multi-scale local descriptor for each point in the point cloud to train a deep classification neural network. Test results from point cloud data collected on 11 highway bridges showed that the proposed method identifies bridge members with the mIoU of 97.72%, significantly higher than 44.29% of PointNet.
In vision-based damage identification, close-up images do not contain the global structure and damage cannot be localized in the global structure. However, the specific location of damage in a full-size bridge structure has an important role to play in bridge health monitoring. Compared to 2D image data, 3D point clouds with extra dimensionality have the ability to model globally but are more difficult to identify small localized damage. Perry et al. 87 linked image-based damage identification results to real-world scale 3D point clouds with the aim of recovering damage size and location information. UAV, computer vision and machine learning are used for the autonomous quantitative detection of bridge damage, while allowing the development of bridge cracks to be tracked. Damage growth was measured using images taken at different times and angles. Finally, the location and extent of the damage was recovered by projecting the 2D image onto a 3D point cloud. Here, the 3D point cloud is mapped from multiple sets of images. In contrast to laser 3D scanning, dense point clouds generated from images captured by UAVs are much less costly. Chen et al. 88 proposed an automated bridge damage assessment method based on low-cost UAV-derived point clouds. 16 million valid point clouds were generated from the 212 images acquired by UAV. Subsequently, the deck regions were extracted using the region growing method and unsupervised learning methods. Finally, a polynomial surface fitting method was used to assess and visualize the damage.
Although point clouds can be used to check the safety of bridges,89–91 point clouds, whether laser scanned or picture mapped, are mostly used for global modeling of structures. When the point cloud corresponding to the overall bridge is used as input to the machine learning network, it is difficult to identify local damage from it. Most of the current research focuses on the autonomous extraction of the individual structural components of bridges. Moreover, point cloud data samples are more difficult to obtain and are available in smaller quantities. Furthermore, the modeling accuracy of the point cloud can be affected when the measuring device is far away from the object. Therefore, it is more appropriate to use images for the detection of locally minor damage. The organic integration of close-up images with 3D point clouds may be a subject of more interest in the future.
Infrared thermography-oriented methods
The combination of images and intelligent algorithms allows for fast bridge damage identification. However, the way in which the image data is collected means that such methods can only identify bridge surface damage. Detecting internal damage to bridge elements has always been a challenge. Thermal imaging technology converts the heat distribution data of a target object into an image by measuring the infrared radiation of the object. 92 Infrared thermography allows humans to see beyond the visual barrier to the temperature distribution on the surface of an object. When damage occurs inside a bridge member, the internal voids are often filled with air or water. Infrared thermography is an alternative imaging method for detecting them and presents a significantly different thermal image than when there is no damage. 93 As a result, thermal images can be used to detect internal damage to a structure, while enhancing the identification of surface damage. The data format of the thermal image is identical to the image. Thus, image-oriented bridge damage identification methods can be very easily transferred to thermal image-oriented damage identification methods. The former includes damage classification, damage identification and damage segmentation, while the latter generally includes only damage identification and damage segmentation. There are few thermal imaging-based damage classification studies.
Damage identification using thermal imaging
As the damaged and undamaged areas have different infrared radiation, infrared thermography can present the damage as a different color from the undamaged area in the thermographic image. Rapid identification of the type and location of damage from thermal imaging images is the main objective of infrared thermography-based bridge damage identification. This type of method still uses the object detection network in image processing. Ali 94 and Ali and Cha 95 used Deep inception neural network and thermal imaging to detect subsurface damage to bridge steel members. He used an uncooled microbolometer to take thermal images of the steel structure on the Arlington Bridge. Inception was used to classify anomalous areas in thermal images achieving a test accuracy of 96%. Meanwhile, ultrasonic pulse velocity tests were used to verify the effectiveness of the proposed method. The combination of infrared technology and deep learning therefore allows autonomous detection of subsurface damage to steel bridge elements with minimal human intervention. In addition, infrared acquisition instruments are easy to install, for example infrared cameras can be mounted on UAVs for quick and easy thermal image acquisition.96,97 Pozzer et al. 98 used deep learning to identify delamination, cracks, spalling and patches in conventional images and thermal imaging respectively. Data test results from a dam and two concrete bridge decks showed that MobileNetV2 performs well in identifying multiple types of damage in thermal images, identifying 79.7% of damage. In contrast to normal images, infrared thermography is not sensitive to light sources. Infrared thermography can overcome the limitations of ordinary images and can be used to identify damage in dark areas or inside structures. Qurishee et al. 99 used Faster R-CNN and infrared thermography for bridge crack assessment. They used Inception V2 as the feature extraction module for Faster R-CNN. The experimental results showed that the proposed method achieved the accuracy of 80.35%. Infrared sensing can therefore be used in tandem with optical sensing, both to detect damage within a structure and to improve the accuracy of surface damage. Jin Lim et al. 100 proposed a vision and infrared thermography-based corrosion detection method for steel bridges. Vision cameras are used to obtain RGB images and infrared cameras to obtain active infrared amplitude images. These two types of images are fused to form a hybrid image. This new hybrid image was fed into the Faster R-CNN to detect corrosion damage on the steel bridge surface and subsurface. The experimental results showed that the mAP of the proposed method is 88.64% and 88.59% for surface corrosion and subsurface corrosion respectively. Therefore, the data fusion method achieves the same accuracy for the identification of subsurface damage as for surface damage. Furthermore, infrared cameras can easily be assembled on UAVs due to their ease of installation. 101 For damage identification, damage localization and classification at the inspection site needs to be done quickly and in real time. As such, an inspection system based on a UAV architecture provides the Internet of Things (IoT) technology to support this approach.
Damage segmentation using thermal imaging
Thermal imaging-based damage identification methods only locate and classify damage in the thermal image, and this method only provides the inspector with rough information about the damage. In contrast to thermal imaging-based damage identification, thermal imaging-based damage segmentation methods require further refinement of damage identification results to support subsequent multi-parameter quantification of damage. Similarly, most of these methods also employ semantic segmentation networks in image processing. Garrido et al. 102 proposed a defect segmentation method based on deep learning and thermal images. An automatic thermal image processing algorithm was used to enhance the thermal contrast between the defective and non-defective parts. Afterward, the defective parts of thermal images were segmented and identified using Mask R-CNN. Furthermore, the visual and thermal images are combined in order to improve the segmentation accuracy. 103 Jang et al. 104 proposed a deep super-resolution segmentation network for assessing bridge cracks. The combination of the two types of images allows a precise evaluation of cracks at the 100 μm. Meanwhile, false alarms due to poor bridge conditions were reduced by 49.91% and 13.31% in accuracy and recall respectively. As a non-destructive testing technique, infrared thermal imaging has a high capability in detecting subsurface delamination. It is less costly, and data can be collected more quickly than other techniques. Cheng et al. 105 proposed a delamination damage segmentation method based on dense convolutional networks and thermal images. Due to the different environments, the size and depth of the bridge deck delamination is somewhat random. For this reason, delamination damage simulation experiments at different depths were used to validate the proposed method. The shape of the delamination identified by the proposed method is highly consistent with the real situation. Omar 106 used a full-size reinforced concrete bridge deck to verify that thermal mosaics generated from a single infrared image can be used by clustering techniques such as K-means for delamination identification. Furthermore, he used fuzzy set theory to explain the uncertainty and inaccuracy in infrared thermal imaging tests.
Compared to visual inspection, intelligent damage detection methods for bridges based on infrared thermal imaging are still at a preliminary stage. After all, infrared cameras are far less common than regular cameras. Moreover, it has certain limitations. In most cases, it is used to inspect bridge elements exposed to direct sunlight. In order to validate the effectiveness of thermal imaging in the delamination inspection of concrete bridges not directly exposed to sunlight, Rocha et al. 107 conducted a series of experiments for evaluation. Polystyrene slabs with different thicknesses and depths and concrete specimens with two water-cement ratios were used to simulate defects in the concrete. Delamination defects in specimens with lower water to ash ratios are easier to detect, and the closer the defect is to the surface, the easier it is to detect. In addition, the ideal time to use a thermal imaging camera is midday. Obscured bridges can prevent the bridge elements from warming up, thus limiting the effectiveness of thermal imaging. Meanwhile, the depth of the damage can also limit the infrared thermography method, and only damage near the surface can be accurately detected. 108 There has been little research into infrared thermography for bridge damage identification as compared to visual inspection. Fewer open-source datasets are also available. The boom in open-source data is somewhat reflective of the research fervor in the field. Ichi and Dorafshan 109 used infrared thermography, impact echoes and ground-penetrating radar to identify bridge deck defects such as cracks, delamination and spalling, respectively. They developed a dataset, namely SDNET2021. These data can contribute to the development of intelligent methods for bridge damage identification based on infrared thermal imaging. Further research is needed to fully exploit the advantages of infrared thermography in bridge damage identification based on its characteristics.
Ground penetrating radar-oriented methods
As a non-destructive detection technique, ground-penetrating radar (GPR) is often used to detect underground targets. Ground-penetrating radar is a method of detecting the properties and distribution of material within a medium by using antennas to transmit and receive high-frequency electromagnetic waves. 110 It has the advantages of fast data acquisition, high resolution and easy operation. Thus, ground-penetrating radar can be used for the identification of internal defects in bridge decks, the layout of reinforcement.111,112 Ground-penetrating radar is better suited to detecting defects deeper inside than infrared thermography. For ground-penetrating radar data, the single point test data is a vector. When the ground-penetrating radar moves, the multi-point test data forms a matrix. Ahmadvand et al. 113 evaluated concrete bridge decks using 1D convolutional neural networks and GPR data. GPR data were collected from eight experimental samples with defect-free and manually created subsurface defects. The proposed method could classify subsurface defects with an accuracy of 84%. In order to improve the identification effectiveness of ground-penetrating radar, some data features are pre-extracted. Kumar et al. 114 obtained properties such as raw mean amplitude, instantaneous amplitude of reflected waves and intensity from the GPR data, which were fed into a random forest model to predict the compressive strength, density and porosity of the material. Data test results collected on real bridges showed that the proposed method is successfully used to identify spatial and temporal variations in concrete properties. All the above studies are directly processing GPR data. However, the multi-point test data is a matrix that reflects the vertical section information of a component. This matrix is often converted to a gray-scale image, that is, a scanned image. Therefore, in this case, the ground penetrating radar-based bridge damage detection method is transformed into the image-based bridge damage detection method.
When ground-penetrating radar data is converted into scanned images, object detection algorithms are mostly used to identify damage or reinforcement in them. Asadi et al. 115 presented a dataset containing 4000 scanned images, named DECKGPRH1.0, obtained from real bridge deck field data. These scanned images are pre-processed with Histogram of Oriented Gradients (HOG) feature descriptor and then fed into AdaBoost to obtain a classifier. A combination of a trained classifier and a sliding window was used to identify and locate the rebar positions. Then, the location of the reinforcement can be used to measure the protective layer thickness of the reinforcement. Hu et al. 116 used a similar method to obtain the steel regions in the scanned images. The difference is that the classification method was random forest. Robust hyperbolas were then used to fit each hyperbolic feature obtaining peaks to locate the reinforcement. Finally, the detected reinforcement, its location and depth are used to create the deterioration map. Real bridge deck-based field tests have shown that the proposed method achieves an accuracy of 98.6%. However, this type of target localization method is implemented by combining a classifier with a sliding window, which is less efficient. In order to improve detection efficiency, a number of classical object detection methods are used. Gao et al. 117 used Faster R-CNN and 3785 scanned images for the identification of structural diseases. The dataset contains three types of defects: reflection cracks, water-damage pits, and uneven settlements. The Faster R-CNN obtained the best identification accuracy when the anchors and ratio were 0.3 and 0.7 respectively. In this case, the identification accuracy of the three defects was 88.31%, 90.56%, and 88.51%, respectively; the IoU was 86.53%, 87.26%, and 84.93%, respectively. Zhang et al. 118 proposed an automatic detection method for bridge deck corrosion based on SSD and scanned images. 10,316 scanned images were used for training and 2578 scanned images were used for testing. The size of each scanned image is 300 × 300 pixels. The trained SSD was used to identify and locate 677 bars in the scanned image with an accuracy of 98%. Generate a contour map of the bridge deck corrosion environment from the extracted reinforcement information. Single sensing data may be limited by the sensing technology and the fusion of multiple data can further improve the identification of damage. Li et al. 119 used YOLO v3 to identify the reflected signals of the reinforcement in the scanned images and to obtain a rough thickness range of the overburden. The obtained thickness range was then fed into a 1D convolutional neural network along with the (Electromechanical impedance) EMI data to estimate the overburden thickness and reinforcement diameter. The sand pit experiments showed an accuracy of 96.8% and 90.3% for the estimation of overburden thickness and reinforcement diameter, respectively.
For the data-driven methods, the identification effectiveness of the subsurface defects in bridge decks depends to a large extent on the data samples. A large number of diverse samples can significantly improve the identification accuracy of machine learning. Ground-penetrating radar is used to collect data on subsurface damage. However, a specific type of damage may be rare in actual engineering inspection, resulting in a small sample of its corresponding data. For most machine learning methods, the smaller the number of data samples corresponding to a specific type of damage, the poorer the identification accuracy of that type of damage. For normal images, a specific type of image can be augmented by random noise, cropping, rotation, etc. However, the scanned image of GPR is highly directional and structured. If the scanned image is rotated or cropped, it will change the characteristics of the scanned image itself. In order to increase the number of scanned images for a specific type of damage, a more intelligent approach to data augmentation is required. Generating adversarial networks (GAN) can be used to learn the deeper features of the original data and thus generate more data with the same type autonomously. Compared to fine-tuned scanned images, this method increases the diversity of samples based on deep-level characteristics. The data samples generated can effectively improve the identification of subsurface defects and objects. 120 However, this type of research is still relatively rare. Further development of the GAN structure and training strategies is required to provide more realistic scanned images.
Vibration response-oriented methods
Almost all of the above methods study bridge damage detection from the point of view of the morphology of the damage as reflected in the various types of sensing data. However, bridge damage does not always show morphological changes. For instance, stiffness degradation is also a type of structural damage and can be identified by the vibration response of the structure. Common structural vibration responses include acceleration, velocity, stress, strain, and displacement, etc. These vibration responses are all one-dimensional vectors at the individual measurement points. As a result, bridge damage detection methods based on one-dimensional vibration signals and machine learning have been widely studied. In recent years, machine learning, especially deep learning, has gained significant results in two-dimensional data such as images. One-dimensional vibration signals are easily converted into two-dimensional signals by pre-processing. Thus, some classical deep network structures can conveniently process two-dimensional signals obtained by pre-processing one-dimensional vibrational signals. As such, this section divides intelligent bridge damage detection methods for vibration response into those based on one-dimensional (1D) vibration signals and those based on two-dimensional (2D) vibration signals.
1D vibration signals-based methods
During bridge damage detection, many studies have converted damage identification into the classification problem. The location, type or degree of the damage is used as a basis for classification. Machine learning algorithms need to learn to identify damage from a large sample of data. In order to improve the sample of data corresponding to each type of damage, finite element models are widely used. Ghiasi et al. 121 used a finite element model to collect acceleration data for a railway bridge in 255 scenarios. 1300 data samples were collected for each scenario. The types of damage were classified as minor, medium and severe. In addition, to further increase the number of samples, Gaussian noise and sine waves were used to complete the data augmentation. 1DCNN can achieve 100% identification accuracy. Parisi et al. 122 obtained strain data using finite element models of steel truss bridges with different damage scenarios. K-nearest neighbors select the most informative features from the strain data, and these features are fed into 1DCNN to achieve 93% accuracy in damage identification. However, these datasets are all derived from finite element models. To further investigate the effectiveness of machine learning algorithms for damage detection in real bridges, some real vibration responses were collected. Akintunde et al. 123 measure the strain response caused by the passage of vehicles at different speeds from a full-size bridge deck. They produced three types of structural damage on the bridge. The first type of damage was caused by a 10-ton monocoque truck making an angled collision with a concrete guardrail at a speed of 93 km/h. The second type of damage is a vertical saw cut at the point of the guardrail collision. The third type of damage is the extension of the saw cut to the deck based on the second type of damage. Then, unsupervised learning methods were used for damage type identification. Delgadillo and Casas 124 used real data from the Warren truss bridge to verify the effectiveness of clustering methods in detecting and locating bridge damage.
Bridge damage detection can also be converted into the regression problem. Deng et al. 125 used support vector machine (SVM) to develop a regression model between fatigue damage and traffic load parameters for suspension bridge hangers. Experimental results showed that the best predictions are obtained when the input is a 4-lane daily traffic flow. Malekjafarian et al. 126 collected vehicle acceleration signals and vehicle speed from a healthy bridge, which were fed into the artificial neural network (ANN) to predict vehicle acceleration signals. Then, Gaussian process is used to detect changes in the distribution of prediction errors. These training data are obtained from a simulated health bridge. Therefore, the prediction error of the trained ANN model increases, indicating that the bridge has been damaged. Bao and Liu 127 summarized the vibration-based bridge scour detection and also analyzed the effects of sensor installation location and scour hole shape on scour detection. However, temperature variations can affect the modal properties of bridges, which in turn can affect vibration-based bridge scour detection. Zheng et al. 128 used a Gaussian process model to establish a probability-based correlation between changes in dynamic properties of the bridge and temperature changes, thus eliminating the effect of temperature changes. The obtained more accurate modal parameters are integrated into a Bayesian inference framework to complete bridge scour detection. Ásgrímsson et al. 129 used a deep neural network with dropout to reconstruct vibration signals from healthy states and output uncertainty intervals. It is rejected when the reconstruction error of the unknown sequence exceeds a threshold. When a sufficient number of sequences have been rejected, the bridge is in a damaged state. The well-known Z24 bridge benchmark dataset was used to validate the proposed method. However, the uncertainty in obtaining the model using dropout came from Gal and Ghahramani’s article. 130 Their approach provides a very simple way for deep learning to obtain uncertainty. This way of obtaining model uncertainty has been controversial. Osband 131 analyzed the risks and uncertainties in deep learning. Risk represents the inherent volatility of the decision outcome, while uncertainty represents confusion about possible outcomes. Risk is a fixed attribute of the problem and cannot be removed by collecting more data. Uncertainty is a property of beliefs and can be cleared by more data. Osband argued that the posterior obtained by dropout is not to be regarded as a Bayesian approximation. Therefore, this approach requires further analysis and discussion. Bayesian theory combined with deep learning has greater potential. Weinstein et al. 132 used ANN and bridge strain data to generate the probabilistic model of bridge behavior. In contrast to the fastest descent method and Newton’s method, they used the Levenberg-Marquardt (LM) algorithm with Bayesian regularization to train the ANN. This method can closely capture the underlying data trends to identify bridge damage. Further organic integration of Bayesian theory with machine learning methods will be an important research topic.
Bridge damage detection can also be converted to anomaly detection. Anomaly detection is often achieved using unsupervised learning methods or clustering methods. The anomaly detection or novelty detection method aims to distinguish normal data from anomalous data due to possible damage. Soleimani-Babakamali et al. 133 proposed an unsupervised novelty detection method based on GAN. GAN is used to address the dynamic class novelty detection barrier while the framework can be adapted to be free of a priori information. GAN and one-class joint Gaussian distribution models are used together to detect novelty. The experimental results showed that the proposed method has a high detection rate in low false alarm conditions. Modal data is one of the most widely used vibration characteristics in bridge monitoring. Modal data extracted from the vibration response can be used for continuous or long-term structural health monitoring. Ensemble learning can be combined with modal information for early damage detection of bridges. An ensemble learning consisting of three main levels was used separately with different forms of Mahalanobis distance metrics in order to improve damage detectability. 134 The kernel null space algorithm maps the modal data to the kernel feature space and projects it to individual points in the null space. 135 It can then be combined with probabilistic threshold estimation to accomplish bridge damage detection under strong environmental changes. Extreme value theory is a probabilistic approach that can be used in novelty detection by considering only some important data points for analysis and modeling. Probabilistic methods enable two important tasks in novelty detection based on modal data in a single framework: decision making and threshold estimation. 136 In addition, active learning can significantly improve the detection and classification performance of bridge damage. 137 However, Mahalanobis-squared distance (MSD)-based anomaly detection is not sufficiently robust when the modal data of structures is affected by environmental or operational variability. Therefore, robust anomaly detection methods need to be studied intensively. Sarmadi and Karamodin 138 proposed an anomaly detection method based on adaptive MSD and one-class KNN rule. The generalized extreme value distribution was modeled using the block maxima method to determine an accurate threshold limit. The experimental results showed that the total error of the proposed method in damage detection was less than 1.86%. Entezami et al. 139 proposed a novel non-parametric anomaly detection method based on empirical machine learning theory. Some empirical measurements and minimum distance values were used to define the new damage index. In addition, the proposed method has the lowest false detection rate and computation time. In anomaly detection, the choice of the threshold value is important. Probabilistic methods under semi-parametric extreme value theory can estimate a quantile from some extreme samples as a threshold. 140 To reduce the impact of environmental variability, the one-class nearest neighbor search-based unsupervised feature selection concept is used to select relevant features and remove irrelevant features that are affected by various variabilities. 141 Subsequently, the enhanced local MSD could be used to determine the distance values or anomaly indices.
2D vibration signals-based methods
In recent years, with the rapid development of deep learning in the field of image processing, a large number of network structures applicable to two-dimensional data have been intensively studied. These network structures have been shown in a large number of studies to have excellent classification and regression capabilities. In order to apply these efficient new networks to vibration response-oriented damage detection methods, one-dimensional signals can be converted into two-dimensional signals by various processing methods. The original structural response signal is reconstructed by wavelet packet filtering to obtain recurrence maps for different damage scenarios, which are used as input to the 2DCNN. He et al. 142 showed that VGG16 and recursive maps can be used to achieve structural damage localization with 100% accuracy. Bao et al. 143 used the time-domain greyscale map of the raw acceleration as an input of 1DCNN to identify the six types of data anomalies from the cable-stayed bridge. In order to satisfy the input of 1DCNN, the two-dimensional gray-scale map was first expanded into an image vector. Experiments showed that the proposed method can achieve a global accuracy of 87%. Although the initial input to this method is a two-dimensional signal, the identification network is still a one-dimensional network. Tang et al. 144 directly plotted the time and frequency domains of the raw acceleration in a single image as input to the 2DCNN. They designed a lightweight 2DCNN with only five layers that can accurately classify seven patterns of data anomalies from large-span cable-stayed bridges. Chen et al. 145 transformed the vibration signal into a power spectral density transmission (PSDT) map that was fed into the 2DCNN for automatic frequency extraction. Mao et al. 146 proposed a data anomaly detection method based on GAN and autoencoder. The vibration response collected from bridges is first converted to the Grammy Angular Field (GAF) image. A generative adversarial network containing a generator and a discriminator is used to learn to generate realistic GAF images. The input and output of an autoencoder consisting of an encoding structure and a decoding structure are both GAF images. For better output for GAF images, the weights of the decoding structure are derived from the generator in GAN. The training data are all normal data. Therefore, when the difference between the input and output of the autoencoder exceeds a threshold value, the input data is determined to be abnormal.
The one-dimensional vibration signal is converted into a two-dimensional signal and presented as an image. However, this type of image is not the same as a photographed image, as it is far less complex than a camera image. Therefore, the direct use of deeper classical networks can cause problems such as gradient disappearance, gradient explosion, or over-fitting. To solve this problem, it is common to simplify classical networks or design lightweight neural networks of their own. In addition, the more critical issue is that of signal conversion. The focus is on converting one-dimensional signals to two-dimensional signals and fully representing the more important information. Cheng et al. 147 and Liao et al. 148 analyzed the effects of the wavelet transform, Hilbert–Huang transform, and Mel frequency cepstral coefficient (MFCC) extraction methods on the identification effectiveness of machine learning methods. The experimental results showed that both the wavelet transform and MFCC make the machine learning method achieve good results. In addition, their results showed that shallow CNNs achieve optimal classification results. Therefore, in this section, the requirement for a deep network should also be analyzed and selected according to the specific engineering problem.
Other types of data-oriented methods
Other than the five types of data-oriented bridge damage detection methods described above, there are other types of data that can be used to identify bridge damage, such as acoustic emissions, satellite data, tabular data, and text data, etc. Xin et al. 149 proposed a bridge cable damage identification method based on deep transfer learning and acoustic emission signals. The acoustic emission signal was processed by wavelet analysis to obtain a two-dimensional RGB time-frequency map, which was fed into a pre-trained Google-Net model for parameter updating and prediction. The experimental results showed that the proposed method achieves the classification accuracy of 99.05%. Satellite data is instantaneous in detecting bridge safety. In particular, the satellite-borne Aperture forming radar (SAR) can be used to extract the modal features of bridges such as natural frequencies and mode shapes even in very harsh weather conditions. 150 Entezami et al. 151 proposed a hybrid unsupervised learning and satellite image-based method for early damage detection of bridges. Extracted displacement samples from satellite images were enhanced and fed into the autoencoder for the reconstruction of the input. The residual matrix of inputs and outputs was used to calculate the novelty score, which is greater than a threshold value indicating the presence of damage. Giordano et al. 152 proposed a bridge damage detection method based on Differential SAR Interferometry (DInSAR) measurements. DInSAR determines the displacement change by the phase difference between two SAR images. Principal component analysis (PCA) was used to remove the influence of the environment, and then novelty detection based on control charts was used to identify damage. Nguyen and Dinh 153 used tabular data and ANN to classify the condition levels of bridges. They extracted data from the NBI database for 2572 bridges including nine parameters: current bridge age, average daily traffic, design load, main structure design, approach span design, number of main spans, average daily truck traffic percentage, average daily traffic growth rate and bridge condition level. The first 8 parameters were used as inputs to the ANN and the last parameter was used as the output of the ANN. The results showed that ANN was able to achieve a prediction accuracy of 73.6%. In addition, the sensitivity analysis results for each input parameter indicated that bridge age is the most significant factor influencing the condition of the bridge, followed by the design load and the main structure design. Similarly, Lim and Chi 154 used XGBoost and tabular data to estimate bridge damage levels. The difference is that they extracted 142,439 inspection records corresponding to 2388 bridges from the Korean Bridge Management System (KOBMS). The extracted data had 53 variables including 34 identification factors, 24 structural factors, 2 traffic factors, and 3 inspection factors. Among these variables, the main structure type, damage type and damage condition level were used as predictive targets. They created 42 decision trees to explain all possible combinations of the 6 types of major structures and 7 types of damages. The experimental results showed that the bridge age is the most important factor, and live load and constant load are the key factors. Li and Burgueño 155 developed four neural networks to predict damage states of bridge abutments. They extracted 1 damage indicator of bridge abutment and 10 variables from the NBI. The abutment damage indicator is Abut_Rating and is assigned 10 integer values from 0 to 9. The smaller the indicator, the more serious the damage to the abutment. The test results showed that ANN can be used to derive the life deterioration curve of the abutment. Shakir Ali et al. 156 used ANN and four parameters to predict the local scour depth of bridge piers. These four parameters are pier diameter, flow depth flow velocity, critical sediment velocity, and sediment particle size. It was found that the bridge pier diameter had a significant effect on the scour prediction results. Garg et al. 157 used factors such as earthquake magnitude, distance from the epicenter to the bridge, bridge age, bridge type and material type to predict whether the bridge would be damaged. The used data was collected from real bridges and contained a total of 250 samples. They compared the classification results of decision trees, random forests, XGBoost and KNN, respectively. Among them, decision trees and random forests obtained the best classification accuracy of 96%. Therefore, the establishment of a large bridge management system, which will contain as much bridge information and inspection data as possible, can provide big data support for intelligent algorithms. Moreover, in the current inspection system, the inspection information is more often included in the inspection report. Moon et al. 158 used Recurrent Neural Network (RNN) with active learning to identify bridge damage from inspection reports. The proposed method could extract damage factors and causal relationships from reports. Furthermore, active learning can effectively reduce the number of labeled data samples, and the F1 score of the model trained by active learning was 0.778 after using only 140 sentences. There are many types of data associated with bridge inspection. The characteristics of each type of data need to be further fully explored and suitable intelligent algorithms selected to provide more accurate identification of bridge damage.
Challenges and future trends
In recent years, artificial intelligence has made a significant contribution to the field of image processing. Many well-known network structures have been designed to solve the problem of target detection in images. Therefore, these target detection methods are directly transferred to bridge damage detection for classification, identification, and segmentation of bridge surface damage in images. However, these network structures are extremely flexible and expandable. Based on these classical target detection networks, new types of networks with other functions such as GANs have been gradually developed. These network structures are highly capable of learning. However, the image data has strong limitations in bridge damage detection. To improve the identification of other types of damage, other sensing data are used in combination with these classical network structures. This combination is divided into two forms: one is the transformation of other sensing data into two-dimensional form, such as infrared thermal sensing, ground-penetrating radar, and two-dimensional vibration signals. The other one only borrows the form of classical networks and modifies the internal components to match the corresponding sensing data, such as 3D point clouds and 1D vibration signals. As the combination of the two continues to deepen, new network structures with their own characteristics have gradually evolved. In traditional bridge damage detection, most of them are oriented toward vibration response. Artificial intelligence techniques have greatly improved the ability of other sensing technologies to be used for bridge damage detection.
Due to the outstanding performance of artificial intelligence algorithms in various fields, various sensing technologies are used in combination with these algorithms for bridge damage detection. Different sensing technologies have greatly improved the detection range of bridge damage and provide high-quality data for artificial intelligence algorithms. However, there are still some challenges with the current intelligent bridge damage detection:
Dataset. The data used in these studies are mini. Mini is reflected in two points: one is that the number of samples is very small, usually containing anywhere from a few 100 to a few 1000 samples. This can hardly be called big data. The other is that the diversity of the samples is poor, and the data used are almost all derived from only one structure. Further, most of the data comes from finite element models, which often deviate from the real structure.
Multi-sensor fusion. Current structural health monitoring systems often have multiple types of sensors already installed. However, most of these papers used only one type of data for their research and analysis, and do not adequately integrate the multiple types of sensing data available.
Fast machine learning methods with incremental updates. Most of the current intelligent algorithms have a large number of training parameters, resulting in long training times. Meanwhile, the structure of most intelligent algorithms is fixed before training and cannot be updated incrementally. As a result, when the dataset is updated, the previously trained network model is discarded, and the network model needs to be retrained. This leads to an increase in training costs and model update times.
Uncertainty estimation of prediction results. Current intelligent algorithms, such as deep learning, only output prediction results and are unable to estimate the uncertainty of the prediction results. The prediction results of intelligent algorithms are not always reliable. Therefore, the uncertainty quantification of the prediction results can assist in making optimal decisions.
Based on these challenges, it is necessary to further in-depth research into intelligent bridge damage detection methods. First, there is an urgent requirement to establish a large, unified bridge damage dataset to provide real big data for bridge damage detection. Second, further research is warranted on intelligent detection algorithms with multi-sensor fusion. Third, there is a huge need to develop fast machine learning to improve training efficiency and reduce the inference time of models. Furthermore, damage detection methods that are dynamically and incrementally updatable are necessary to satisfy the practical engineering needs. Finally, the organic integration of machine learning with Bayesian theory will be a key topic in order to estimate the uncertainty of the prediction results.
Conclusion
This paper summarizes the artificial intelligence-based methods for bridge damage detection from the perspective of data types. In recent years, artificial intelligence algorithms, especially deep learning, have achieved remarkable results in the field of image processing. As a result, image and artificial intelligence-based methods for bridge damage detection have received widespread attention and have become a hot topic in the field. Therefore, this paper first introduces image-oriented methods for bridge damage detection. However, image data has its own limitations, as it can only detect two-dimensional damage on the bridge surface. Although artificial intelligence algorithms and image data achieve high accuracy and efficiency in bridge surface damage, they can only perform a portion of bridge damage detection. In order to improve the detection capability of bridge damage, other sensing techniques are also used in combination with artificial intelligence algorithms. The 3D point cloud allows 3D information to be obtained about the structure and facilitates the overall modeling. Infrared sensing and ground-penetrating radar are used to sense shallow internal damage and deep internal damage to structures respectively. Further, not all damage is presented in a specific form. Damage such as structural degradation can be sensed from the structural dynamic’s perspective through the vibration response of bridges. Therefore, this paper overviews bridge damage intelligent detection methods oriented toward image, 3D point cloud, infrared sensing, ground-penetrating radar, vibration response, and other types of data, respectively. However, the data used in these studies are relatively simple, for example, from a single real bridge or a single finite element model. The universality of these intelligent methods needs to be further tested. From an algorithmic perspective, research into intelligent learning algorithms based on small samples is urgent and pressing. In addition, further research into data fusion mechanisms to fully exploit the characteristics of each type of data will be developed to provide more accurate bridge damage detection.
Footnotes
Handling Editor: Chenhui Liang
Declaration of conflicting interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This research has been supported by the Science and Technology Development Fund of the Macau SAR under research grant SKL-IOTSC(UM)-2021-2023; Guangdong-Hong Kong-Macau Joint Laboratory Program under research grant 2020B1212030009 and Science and Technology Project of State Administration for Market Regulation under research grant 2021MK044.
