Deep learning-based obstacle-avoiding autonomous UAVs with fiducial marker-based localization for structural health monitoring

Abstract

This paper proposes a framework for obstacle-avoiding autonomous unmanned aerial vehicle (UAV) systems with a new obstacle avoidance method (OAM) and localization method for autonomous UAVs for structural health monitoring (SHM) in GPS-denied areas. There are high possibilities of obstacles in the planned trajectory of autonomous UAVs used for monitoring purposes. A traditional UAV localization method with an ultrasonic beacon is limited to the scope of the monitoring and vulnerable to both depleted battery and environmental electromagnetic fields. To overcome these critical problems, a deep learning-based OAM with the integration of You Only Look Once version 3 (YOLOv3) and a fiducial marker-based UAV localization method are proposed. These new obstacle avoidance and localization methods are integrated with a real-time damage segmentation method as an autonomous UAV system for SHM. In indoor testing and outdoor tests in a large parking structure, the proposed methods showed superior performances in obstacle avoidance and UAV localization compared to traditional approaches.

Introduction

Deep learning-based structural damage detection is a hot topic in the construction and structural health monitoring (SHM) field. Cha et al.^1,2 proposed a new deep convolutional neural network (DCNN) for structural damage detection using vision sensors. It has received enormous interest worldwide, with numerous studies following.^3–5 To quantify detected damage, damage segmentation methods using deep learning have been extensively investigated by many researchers. For example, U-Net ⁶ and its transformed algorithm have been applied.^7–9 Choi and Cha ¹⁰ suggested a fast, efficient deep semantic network for crack segmentation in complex scenes by using an atrous pyramid pooling network and “DenSep” modules. Saleem et al. ¹¹ applied Mask R-CNN ¹² for crack segmentation. They tested image data of a real bridge structure collected by unmanned aerial vehicles (UAVs). Yu et al. ¹³ developed a DCNN and enhanced chicken swarm algorithm to improve the accuracy and computational cost for the automated detection of concrete cracks. Abdeljaber et al. ¹⁴ developed a novel, fast, and accurate real-time structural damage detection system using one-dimensional convolutional neural networks (1D CNNs) that automatically extract damage-sensitive features from raw acceleration signals. In another study, a new approach for detecting compressive stress and load-induced damage in concrete structures using a DCNN integrated with electromechanical admittance (EMA) was proposed. ¹⁵ Two-dimensional convolutional neural network (2D CNN) for rapid damage quantification in concrete structures using raw EMA data has also been used. ¹⁶ Potenza et al. ¹⁷ suggested color histogram-based segmentation for UAV data collected from a real bridge structure. Kang and Cha ¹⁸ developed a semantic transformer representation network (STRNet) to segment cracks at the pixel level in a complex background scene. It shows state-of-the-art performance in terms of accuracy and processing speed. The reported mean intersection over union (mIoU) is 92.6%, with 49.2 frames per second (FPS) and an image size of 1024 × 512 × 3. However, the method has not yet been implemented with UAV images but rather with images collected from fixed-installed cameras. Bao and Li ¹⁹ proposed that machine learning can be used to analyze monitoring data to discover and model the performance and conditions of a structure, thereby creating a “machine learning paradigm” for SHM. ¹⁹ In another study, a fusion CNN architecture was developed for crack identification in the steel box girders of bridges based on real-world images containing complicated disturbance information. ²⁰

UAVs are attractive for various applications, including SHM, since they can overcome accessibility difficulties without heavy machinery, reduce overall costs, and increase the frequency of inspections.^21,22 For example, McGuire et al. ²² tested a commercial UAV for bridge and tower inspections. Benz et al. ²³ conducted a bridge inspection using a UAV and suggested a new crack segmentation algorithm based on a pretrained CNN. However, all these applications are limited to manual flights. Manually flying a UAV requires a skillful pilot and thus is costly, nor is it feasible for mass application to numerous bridge structures.

Recently, Kang and Cha ²⁴ and Ali et al. ²⁵ developed an autonomous UAV flight method for structural damage detection and localization in areas where a global position system (GPS) signal is not available. Generally, GPS signals are weak or unavailable beneath a bridge deck or indoors. To replace the GPS signal, an ultrasonic beacon system (USB) was used to generate a three-dimensional (3D) pseudo map. Additionally, a deep CNN ²⁵ was used to detect cracks on concrete floors. Kang and Cha ²⁴ study was the first trial of an autonomous UAV system for SHM. However, USB signals are vulnerable to various magnetic fields due to transmission towers or electric vehicles and machinery. Further, the method did not have a function to avoid obstacles in the planned trajectory of autonomous navigation.

Obstacles such as beams or pillars can present difficulties for this UAV-based SHM. There are various solutions to avoid obstacles. For example, González et al. ²⁶ implemented a one-degree laser distance sensor in the frontal direction of a UAV. In their research, the UAV detected a wall by using the distance sensor, but this 1D sensor missed small obstacles or obstacles that are not in field of the (1D) sensor. Song et al. ²⁷ implemented a 3D light detection and ranging (LIDAR)-based simultaneous localization and mapping for obstacle avoidance and 3D mapping for bridge inspection. 3D mapping using 3D LIDAR is promising for future SHM, but 3D LIDAR is much more expensive and much heavier than an RGB camera. Therefore, it may require very large payload UAVs, which are not appropriate for navigating inter-story or narrow areas, such as between pillars or bridge piers.

Although in the SHM field, research on obstacle avoidance methods (OAMs) for UAVs has not yet started, obstacle avoidance for UAVs is a popular topic in robotics. For example, Mori and Scherer ²⁸ suggested one of the first attempts for a single camera-based obstacle avoidance algorithm for a UAV. The algorithm detects the obstacle by using speeded-up robust features ²⁹ and avoids it. More recently, Smolyanskiy et al. ³⁰ and Loquercio ³¹ suggested deep learning-based OAMs by supervised training of a CNN to predict steering angle and collision probability in a road scene image for UAV control, but this approach is not guaranteed to respond to unexpected environmental features because the algorithm is limited to the variety of training data. Afterward, depth estimation from a single camera was developed for obstacle distance estimation.^32,33 However, none of these approaches have been implemented in the SHM field.

In this paper, to overcome the limitations of the USB and realize an obstacle avoidance autonomous UAV system for SHM in a GPS-denied environment, a new OAM based on fiducial markers is proposed. Fiducial markers do not need electric batteries to operate, and small markers (4 cm × 4 cm) can be attached at UAV waypoints to establish a navigation trajectory. The developed autonomous UAV system was tested in indoor and outdoor environments, and the collected videos were analyzed by STRNet to segment cracks at the pixel level. The paper is organized into three additional sections. Section “Methodology” describes the proposed OAM with a fiducial marker-based autonomous navigation method and provides a brief explanation of STRNet. In Section “Case studies,” the experimental tests and performances of the OAM and STRNet are discussed. Section “Conclusion and future direction” presents the conclusion and limitations and suggests future improvements.

Methodology

An obstacle-avoiding autonomous UAV navigation control method is proposed for civil SHM. The automated navigation method is composed of a two-level fight controller using two proportional -integral -derivative (PID) controllers and an object avoidance method to modify the planned trajectory of the UAV system when the UAV encounters obstacles, as shown in Figure 1. We propose two types of applications for the OAM, namely 3D USB-based (https://marvelmind.com/) and fiducial ArUco marker-based ³⁴ (https://www.uco.es/investiga/grupos/ava/portfolio/aruco/.) localization techniques to realize autonomous navigation in GPS-denied areas (Figure 1).

Figure 1.

Autonomous UAV-based SHM system architecture. (a) 3D-USB-based autonomous navigation and (b) ArUco marker-based autonomous navigation.

To inspect civil infrastructure, as the first step, waypoints of the UAV navigation should be defined using either 3D USBs or ArUco markers. The planned waypoints are a series of 3D coordinates composed of $x_t^p,y_t^p,z_t^p$ , and ${\theta }_t^p$ , which stand for ground longitudinal coordinate, transverse coordinate, vertical (height) coordinate, and direction of UAV (yaw), respectively. p and t stand for planned waypoints and time steps during the navigation, respectively. The first PID controller (PID_1) calculates the required pitch $\left(P_t\right)$ , yaw $\left(Y_t\right)$ , roll $\left(R_t\right)$ , and throttle $\left(T_t\right)$ to achieve the planned waypoints $(x_t^p,y_t^p,z_t^p,{\theta }_t^p)$ based on the discrepancy with the current measured coordinates $(x_t^m,y_t^m,z_t^m,{\theta }_t^m)$ . In the proposed OAM, if an obstacle is detected, the OAM generates a new waypoint (i.e., $x_t^n,y_t^n,z_t^n,{\theta }_t^n$ ) in each time step to fully avoid it.

For the first approach (Figure 1(a)), a 3D pseudo map is constructed using four stationary beacons in the area the UAV will navigate. The measured coordinates $(x_t^m,y_t^m,z_t^m)$ indicate the actual location of the mobile beacon mounted on the UAV, which is localized by calculation of the 3D distances compared to the four stationary beacons. The yaw angle, ${\theta }_t^m$ , is calculated by the magnetic compass sensor inside the UAV. The calculated $P_t$ , $Y_t$ , $R_t$ , and $T_t$ are inputted to PID_2, which is installed in the UAV to generate a current for each motor of the wings of the UAV. Two cameras are installed on the UAV: one (Camera_1) for front monitoring and the other (Camera_2) for downward floor inspection. The video images, $I_t^{m1}$ , from Camera_1 are processed by an obstacle avoidance algorithm, and if there is an obstacle, a new waypoint $(x_t^n,y_t^n,z_t^n,{\theta }_t^n)$ is determined to avoid collision. The other video images, $I_t^{m2}$ , from Camera_2 are analyzed through a pixel-level segmentation method (i.e., STRNet).

For the second approach (Figure 1(b)) of localization, ArUco markers are used to create a 3D-pseudo map in the area the UAV will navigate. This method detects markers in the incoming image frame $I_t^{m2}$ and decrypts them to find the imbedded binary code of each marker in the marker bank, which has all the information of each marker’s number and coordinates in the pseudo map. Using these decrypted marker numbers and coordinates, the UAV’s position in the 3D-pseudo map $(x_t^m,y_t^m,z_t^m)$ and current yaw ${\theta }_t^m$ are calculated for UAV localization (See Section “ArUco marker detection”). These measured values are given as input to PID_1, which calculates the desired control values $P_t$ , $Y_t$ , $R_t$ , and $T_t$ . More details of the obstacle avoidance algorithm, ArUco marker detection module, UAV localization, and STRNet are provided in the subsections.

Obstacle avoidance method

The overall procedure of the obstacle avoidance algorithm that we propose in this paper (as shown in Figure 2) is as follows: (1) obstacles are detected by You Only Look Once version 3 (YOLOv3) ³⁵ using the input image, $I_t^{m1}$ , in Figure 1; (2) the YOLOv3 (https://pjreddie.com/darknet/yolo/) provides the bounding boxes to localize the obstacles within the image; and (3) if there are multiple bounding boxes for multiple obstacles, box clustering should be done first using the standard deviation (SD) filter and K-means clustering method. This box clustering provides multiple box groups, and decision-making should be done to avoid the closer obstacle.

Figure 2.

Steps in obstacle avoidance.

In this study, the UAV can avoid the obstacle by moving either of right or left direction. The decision is made based on the location of the center point of the clustered bounding box compared to the center point of the entire image, $I_t^{m1}$ . Each step is detailed below.

Detection of obstacle using YOLOv3

To detect any number of obstacles, YOLOv3, which is composed of an encoder and decoder, is used as an example network. It can be replaced by any recent version of YOLO series. DarkNet53 ³⁶ is used as the encoder. The decoder of the YOLOv3 consists of three detection layers (expressed as greenish boxes in Figure 3) and upsampling layers having different scales (expressed as blueish boxes in Figure 3).

Figure 3.

YOLOv3 architecture.

The input size of YOLOv3 is resized to 416 × 416 × 3 from the original image size of 540 × 480 × 3. DarkNet 53 is used to extract the feature maps having sizes of 13 × 13 × 255, 26 × 26 × 255, 52 × 52 × 255 in “Detection Layer 1,”“Detection Layer 2,” and “Detection Layer 3,” respectively, as shown in Figure 3. The main purpose of using these multiple feature maps by upsampling is to consider different levels of features to improve the object (obstacle) detection performance. The depth of the feature map is evenly 255, composed of three anchors (i.e., anchor 1, anchor 2, and anchor 3). Each anchor dimension is 1 × 1 × 85 to represent bounding box coordinates $(b_x,b_y,b_w,b_h)$ , object confidence, and class scores $(C_1,\cdots ,C_n)$ , as shown in Figure 4.

Figure 4.

Obstacle detection process.

As an example, multiple chairs are used as obstacles. The bounding box coordinates, object confidence, and class scores are multiplied by the sigmoid function, and the bounding box width, $b_w$ , and height, $b_h$ , are multiplied by nine predefined box ratios, which can be found in the original paper. ³⁵ The nonmaximum suppression algorithm is applied to remove the overlapped same-class object. When there are two bounding boxes having the same class, and the boxes are overlapped by more than 50%, the bounding box with a lower-class score is deleted.

Obstacle clustering

YOLOv3 provides N number of bounding boxes depending on the number of obstacles in the input image, where $N=\{1,2,3\cdots n\}$ . The resulting bounding box coordinates $[(b_x^1,b_y^1)\cdots (b_x^n,b_y^n\left)\right]$ with dimensions $[(b_w^1,b_h^1)\cdots (b_w^n,b_h^n)]$ are inputted to the obstacle clustering method that we propose in this paper, as shown in Figures 2 and 5.

Figure 5.

Box clustering algorithm.

As the first step of the method, a “Size filter” (i.e., $b_w*b_h>\alpha$ ) is proposed to remove extremely small boxes, which can be assumed to indicate that the object is too small or too far from the UAV. The next step is to determine the distribution level of the obstacles. An “standard deviation (STD) filter” is formulated as Equation (1).

$$\mathrm{std}[(b_y^1+\frac{b_h^1}{2})\cdots (b_y^n+\frac{b_h^n}{2})]>\beta$$

(1)

where $b_y^n+\frac{b_h^n}{2}$ is the size index of the object and the distance of the object from the UAV. When the calculated STD value is smaller than the threshold (β), it is assumed that there are multiple obstacles as a single group; then, without processing of the K-means clustering, we do “Box grouping,” as shown in Figure 5. The threshold values, $\alpha ,\mathrm{and}\beta$ , are defined as 1500 and 20, respectively; these can be determined by the user’s engineering judgment by considering the flight environment and possible obstacle sizes. The range of α can be set between 0 and the total number of pixels in the image. During the YOLOv3 testing for different obstacle configurations, we observed that all significant obstacles were represented by a bounding box, where $b_w*b_h>1500$ . Therefore, smaller bounding boxes were noise or nonsignificant obstacles to improve processing speed in the real-time obstacle avoidance algorithm. Similarly, the β hyperparameter was set based on the testing results of images using YOLOv3. We observed using trial and error that 1500 and 20 were optimal values for α and β, respectively, to avoid obstacles and maintain efficient and safe real-time avoidance of obstacles.

When the calculated STD value is greater than β, K-means clustering is applied to form multiple groups. This K-means clustering generates multiple groups to the bounding boxes that are provided by the YOLOv3, as shown in Figure 6.

Figure 6.

Obstacle detection and obstacle clustering: (a) original YOLOv3 result and (b) obstacle clustering result.

The output from K-means is $[[b_{xG}^1,b_{yG}^1,b_{wG}^1,b_{hG}^1]\cdots [b_{xG}^n,b_{yG}^n,b_{wG}^n,b_{hG}^n\left]\right]$ , where $G=1,2,\dots ,g$ is the assigned group of the bounding boxes. Finally, the bounding box size and coordinates of each group are determined by the box grouping formulation expressed in Equation (2).

$$\begin{array}{l}{\stackrel{.}{b}}_{xG}=\min \left[b_{xG}^1\cdots b_{xG}^n\right]\\ {\stackrel{.}{b}}_{yG}=\min [b_{yG}^1\cdots b_{yG}^n]\\ {\stackrel{.}{b}}_{wG}=\max \left(b_{xG}^1,\cdots ,b_{xG}^n\right)-{\stackrel{.}{b}}_{xG}\\ {\stackrel{.}{b}}_{hG}=\max (b_{yG}^1,\cdots ,b_{yG}^n)-{\stackrel{.}{b}}_{yG}.\end{array}$$

(2)

Using the calculated values $({\stackrel{.}{b}}_{xG},{\stackrel{.}{b}}_{yG},{\stackrel{.}{b}}_{wG},{\stackrel{.}{b}}_{hG})$ from Equation (2), we can classify the obstacles into distinct groups (red- or blue-colored outlines of bounding boxes are shown in Figure 6(b))

Estimation of distance of UAV from obstacle

Now, we should calculate the distance between the UAV camera and the obstacle to figure out how to avoid it. For estimation of the distance of the UAV from the obstacle, it is assumed that width W and height H of the obstacle are known because YOLOv3 is used to predict the bounding box and class of obstacle. To increase the robustness of our algorithm and adapt it to different types of obstacles, we logged the heights and widths of various obstacles that may be present in the SHM environment. To obtain the distance of the object from the camera, the pinhole camera model is used, as shown in Figure 7.

Figure 7.

Schematic view of a pinhole camera model.

In the first step of distance calculation, the obstacle is detected in image $I_t^{m2}$ (see Figure 1) in the form of a bounding box with dimensions $({\stackrel{.}{b}}_{xg},{\stackrel{.}{b}}_{yg},{\stackrel{.}{b}}_{wg},{\stackrel{.}{b}}_{hg})$ as shown in Figure 6(b). For distance calculation, first, the projection of the obstacle’s bounding box on the camera sensor is calculated, as shown in Equations (3) and (4).

$${\stackrel{·}{b}}_h^s\left(\mathrm{mm}\right)=\frac{{\stackrel{·}{b}}_h\left(\mathrm{px}\right)}{H\left(\mathrm{px}\right)}\times S_h\left(\mathrm{mm}\right),$$

(3)

$${\stackrel{·}{b}}_w^s\left(\mathrm{mm}\right)=\frac{{\stackrel{·}{b}}_w\left(\mathrm{px}\right)}{W\left(\mathrm{px}\right)}\times S_w\left(\mathrm{mm}\right),$$

(4)

where ${\stackrel{·}{b}}_h^s$ and ${\stackrel{·}{b}}_w^s$ are the height and width of the bounding box projection on the camera sensor in mm, respectively. $S_h$ and $S_w$ are the height and width of the camera sensor provided in the technical specifications of the camera, respectively. Moreover, using Zhang’s calibration technique, ³⁷ the camera’s intrinsic parameters including focal length $F_l$ are calculated. Using geometric equalities from Figure 7, $d_o$ can be calculated using Equation (5) ³⁸ :

$$d_o=\frac{h_o}{\cos \left(\theta \right)}\times \frac{F_l\left(\mathrm{mm}\right)}{{\stackrel{·}{b}}_h^s\left(\mathrm{mm}\right)}$$

(5)

where $\theta$ is the tilt of camera with respect to the horizontal plane and $h_o\cos \left(\theta \right)$ is the plan of the object perpendicular to the camera sensor plan, as shown in Figure 7.

Generating new waypoint

When obstacles are detected in Step 1 in Figure 8, the proposed OAM has to figure out a new waypoint based on the distance between the UAV and the obstacles. In this case, only the first group of obstacles closer to the UAV (shown in Figure 6(b) as a red rectangle) is considered. When the calculated distance $d_o$ between the nearest obstacle and the UAV is less than a threshold distance, the OAM assumes that the obstacle is close enough to the UAV, and the UAV is stopped automatically at its current location to avoid the collision. Through experiments, the threshold distance value can be adjusted between 1 m and 2 m depending on safety concerns.

Figure 8.

Generating new waypoints with OAM.

The next step is to determine a new waypoint to avoid obstacles. To do this, the center $(\frac{W}{2},\frac{H}{2})$ of the image $I_t^{m2}$ and the center $(\hat{x},\hat{y})$ of the bounding box are compared. If $(\frac{W}{2}-\hat{x})$ is positive, as shown in Figure 8, the new waypoint is generated at the right side with a 1 m distance, that is, $(x_t^n,y_t^n-1m,z_t^n)=(x_{t+1}^n,y_{t+1}^n,z_{t+1}^n)$ in Step 2, as shown in Figure 8. Here, $\frac{W}{2}$ is the horizontal center of the field of view. This rule can be expressed as shown in Equation (6)

$$x_{t+1}^n,y_{t+1}^n,z_{t+1}^n=\{\begin{array}{c}(\frac{W}{2}-\hat{x})>0:x_t^n,y_t^n-1m,z_t^n\hfill \\ \multicolumn{1}{c}{(\frac{W}{2}-\hat{x})\le 0:x_t^n,y_t^n+1m,z_t^n}\end{array}$$

(6)

If $[\frac{W}{2}-({\stackrel{·}{b}}_x+{\stackrel{·}{b}}_w)]\le 0$ , the UAV should be moved by one more meter to the right (along y axis). This should be continued until $[\frac{W}{2}-({\stackrel{·}{b}}_x+{\stackrel{·}{b}}_w)]>0$ . If the new waypoint is sufficient $[\frac{W}{2}-({\stackrel{·}{b}}_x+{\stackrel{·}{b}}_w)]>0$ to avoid the obstacles, then another waypoint $(x_{t+2}^n,y_{t+2}^n,z_{t+2}^n)$ is generated $=(x_{t+1}^n+1m,y_{t+1}^n,z_{t+1}^n)$ as Step 3 in Figure 8. Equation (7) describes the two cases of decisions for Step 3, in which the UAV avoids the left or right obstacles from Step 2

$$\begin{array}{c}{x}_{t+2}^n,y_{t+2}^n,z_{t+2}^n=\mathrm{Left}\hfill \\ \multicolumn{1}{c}{\{\begin{array}{c}\frac{W}{2}-{\stackrel{·}{b}}_x<0:x_{t+1}^n+1m,y_{t+1}^n,z_{t+1}^n\hfill \\ \multicolumn{1}{c}{\frac{W}{2}-{\stackrel{·}{b}}_x\ge 0:x_{t+1}^n+1m,y_{t+1}^n+\mathrm{xm},z_{t+1}^n}\end{array}}\\ x_{t+2}^n,y_{t+2}^n,z_{t+2}^n=\mathrm{Right}\hfill \\ \multicolumn{1}{c}{\{\begin{array}{c}\frac{W}{2}-({\stackrel{·}{b}}_x+{\stackrel{·}{b}}_w)<0:x_{t+1}^n+1m,y_{t+1}^n-\mathrm{xm},z_{t+1}^n\hfill \\ \multicolumn{1}{c}{\frac{W}{2}-({\stackrel{·}{b}}_x+{\stackrel{·}{b}}_w)\ge 0:x_{t+1}^n+1m,y_{t+1}^n,z_{t+1}^n}\end{array}}\end{array}$$

(7)

At the new waypoint $(x_{t+2}^n,y_{t+2}^n,z_{t+2}^n)$ , the obstacles are disappeared from the image frame; then, the UAV moves to the originally planned waypoint $(x_{t+3}^p,y_{t+3}^p,z_{t+3}^p)$ to return to the originally planned trajectory. All these processes are repeated until the end of the planned navigation.

ArUco marker detection

To navigate a GPS-denied area autonomously, a USB was implemented by Kang and Cha. ²⁴ However, the USB was vulnerable to magnetic interference; therefore, sometimes, it did not provide reliable signals and resulted in poor localization of the UAV, which is explained in later sections. To overcome this limitation of the USB, we introduce an ArUco marker-based UAV localization method.^34,39 An ArUco marker is a square-shaped marker, as shown in Figure 9, with a black and white square patch containing a unique barcode. Multiple numbers of installed ArUco makers establish a 3D pseudo map to replace GPS signals, and each ArUco marker’s 3D coordinates based on this 3D pseudo map are saved in a marker library in the ground station. Then, we can set the starting waypoint of the UAV based on this 3D map. Because each marker has a unique binary barcode, the barcode number is used to find its location information in the library; therefore, we must read the barcode using image processing.

Figure 9.

Schematic block diagram of ArUco marker detection process: (a) convert image to grayscale, (b) local adaptive thresholding and contour filtering, (c) rectangular image patch selection, and (d) assign the binary value 1 or 0.

The first step is to detect this barcode from $I_t^{m2}$ , as shown in Figure 1. For this purpose, the image $\left(I_t^{m2}\right)$ is converted to grayscale to extract the barcode, as shown in Figure 9(a). The grayscale image is binarized using local adaptive thresholding to extract the edges of the object in the frame, as shown in Figure 9(b). After that, contours are extracted from the binarized image by using Suzuki and Abe’s algorithm, ⁴⁰ which produces many contours, most of which are irrelevant to marker detection. We only need rectangular contours for marker detection because of the marker’s shape; therefore, using the Douglas–Peucker algorithm, ⁴¹ polygonal approximation is performed, and contours with four corners are selected for further processing. Then, a valid rectangular contour patch is extracted from the grayscale image, as shown in Figure 9(c). The extracted rectangular image patch should be converted to a true black-and-white image by using Otsu’s method. ⁴² Then, the rectangular image patch is divided into equally sized 7 × 7 grids, as shown in Figure 9(d). For all black grids, 0 value is assigned, and for all white grids, 1 value is assigned (see Figure 9(d)). Using these 7 × 7 grids, only the core 5 × 5 grids are used to read the barcode of the marker to determine the number (ID) of the marker. The marker’s ID in the marker library is used to find the coordinates of the detected marker. So, each marker’s coordinates should be predefined based on the location of each marker within the 3D pseudo map. Using the coordinates of the detected marker, the current location of the UAV can be calculated within the 3D pseudo map, as is explained in detail in the next section.

The ArUco marker detection method is robust against environmental distractions by using unique corner detection and increasing the number of bits in the marker. Using markers with a higher number of bits ensures that the pattern is complicated, so that it does not occur in the monitoring environment and improves robustness. Additionally, false detections can be filtered by using a predefined dictionary of markers. If a fake marker is detected and it exists in the dictionary, it will not affect the UAV’s position as it corrects its position based on the next true marker detected.

UAV localization

For autonomous flight using only ArUco markers, UAV spatial coordinates and pose $(x_t^m,y_t^m,z_t^m,\mathrm{and}{\theta }_t^m)$ in the pseudo map must be calculated from the incoming image $I_t^{m2}$ . For this localization, the first step is to create a 3D pseudo map by installing markers, which are the waypoints in the flight zone. Figure 10 shows an example of attached ArUco markers (depicted as white circles) as the waypoints for the UAV to generate a planned trajectory of the flight.

Figure 10.

Attached ArUco markers (white circle) in the ceiling of the reinforced concrete structure.

Each marker in the pseudo map has its own local coordinate system ${\mathit{O}}_A(x_{\mathrm{Ai}},y_{\mathrm{Ai}},z_{\mathrm{Ai}})$ having an origin at the center of the marker where i stands for the ith marker in the pseudo map, as shown in Figure 11. Similarly, a UAV has its own local coordinate system ${\mathit{O}}_d(x_d,y_d,z_d)$ . The centroidal axis of the UAV body is $x_d$ , horizontal direction is $y_d$ , and vertical direction is $z_d$ . The camera coordinate system ${\mathit{O}}_c(x_c,y_c,z_c)$ has its origin at the optical center of the camera, with the x-axis pointing right and the y-axis pointing front, as shown in Figure 11. The pixel coordinate system of the marker image is given by ${\mathit{O}}_f(u_f,v_f)$ with the origin at the top left of the image frame $I_t^{m2}$ .

Figure 11.

UAV localization in the pseudo map using ArUco markers.

In the first step of localization, the marker’s unique barcode information is decrypted from the incoming frame $I_t^{m2}$ as discussed in Section “ArUco marker detection.” The marker library contains four possible orientations of a barcode for each marker, which are used to recognize the corners ${\mathit{M}}_{\mathit{p}}=(M_1,M_2,M_3,\mathrm{and}{M}_4)$ where $\mathit{p}=1,2,3,4$ and assign local coordinates ${\mathit{O}}_A=(x_{\mathrm{Ai}},y_{\mathrm{Ai}},z_{\mathrm{Ai}})$ to the marker, as shown in Figure 11.

Next, to find the location and pose of the camera using ArUco markers, we adopted Burger’s pose of the camera estimation method ⁴³ to this UAV camera pose estimation problem. It is assumed that the markers’ dimensions $(l_A\times l_A)\mathrm{cm}$ in real-world units are known and the marker is on a plane surface. The coordinates of each corner point $M_p={(X_p,Y_p,0)}^T$ of the ${\mathit{O}}_A$ , ArUco coordinate system and its two-dimensional projection in pixel coordinates $m_p={(u_p,v_p)}^T$ are also known. ${\overline{M}}_p={({\overline{X}}_p,{\overline{Y}}_p,0,1)}^T$ and ${\overline{m}}_p={({\overline{u}}_p,{\overline{v}}_p,1)}^T$ are the projective coordinates of $M_p$ and $m_p$ , respectively, for convenience of mathematical calculation. The aim of UAV localization is to find the extrinsic parameters of the UAV camera $[{\mathit{R}}_A^c|{\mathit{T}}_A^c]$ , which is a $3\times 4$ matrix as shown in Equation (8), where ${\mathit{R}}_A^c$ and ${\mathit{T}}_A^c$ are rotation and translation transformation of the ArUco marker compared to the camera coordinate system ${\mathit{O}}_c$ , respectively. These two matrices will eventually be used to determine the spatial coordinates and pose $(x_t^m,y_t^m,z_t^m,\mathrm{and}{\theta }_t^m)$ of the UAV in the pseudo map ${\mathit{O}}_g$ .

$$\left[{\mathit{R}}_A^c|{\mathit{T}}_A^c\right]=\left[\begin{array}{ccc}{r}_{11}& r_{12}& \begin{array}{cc}{r}_{13}& t_x\end{array}\\ r_{21}& r_{22}& \begin{array}{cc}{r}_{23}& t_y\end{array}\\ r_{31}& r_{32}& \begin{array}{cc}{r}_{33}& t_z\end{array}\end{array}\right]=\left[\begin{array}{ccc}{\mathit{r}}_1& {\mathit{r}}_2& \begin{array}{cc}{\mathit{r}}_3& {\mathit{T}}_A^c\end{array}\end{array}\right],$$

(8)

where $r_{\mathrm{ij}}$ is an element of the rotation matrix ${\mathit{R}}_a^c$ , and $(t_x,t_y,t_z)$ are the elements of translation matrix ${\mathit{T}}_A^c$ . To determine the unknown variables of Equation (8), the camera intrinsic parameter matrix A , which can represent optical characteristics of the camera such as focal length, principal point, skewness, etc., should be implemented. A is expressed in Equation (9):

$$\mathit{A}=\left[\begin{array}{ccc}{\alpha }_u\hfill & s\hfill & u_0\hfill \\ \multicolumn{1}{c}{0}& {\alpha }_v\hfill & v_0\hfill \\ \multicolumn{1}{c}{0}& 0\hfill & 1\hfill \end{array}\right],$$

(9)

where ${\alpha }_u\mathrm{and}{\alpha }_v$ are the camera scale factors, $(u_0,v_0)$ is the principal point, and s is the skewness coefficient of the camera. For this purpose, the pinhole camera projection model is used in Equation (10) ³⁷ :

$$\begin{array}{c}\left[\begin{array}{c}{\overline{u}}_p\hfill \\ \multicolumn{1}{c}{\begin{array}{c}{\overline{v}}_p\hfill \\ \multicolumn{1}{c}{1}\end{array}}\\ \multicolumn{1}{c}{\overline{w}}\end{array}\right]=\mathit{A}\left[\begin{array}{cccc}{\mathit{r}}_1\hfill & {\mathit{r}}_2\hfill & {\mathit{r}}_3\hfill & {\mathit{T}}_A^c\hfill \end{array}\right]\left[\begin{array}{c}{\overline{X}}_p\hfill \\ \multicolumn{1}{c}{{\overline{Y}}_p}\\ \multicolumn{1}{c}{0}\\ \multicolumn{1}{c}{1}\end{array}\right]\hfill \\ \left[\begin{array}{c}{\overline{u}}_p\hfill \\ \multicolumn{1}{c}{{\overline{v}}_p}\\ \multicolumn{1}{c}{\overline{w}}\end{array}\right]=\mathit{A}\left[\begin{array}{ccc}{\mathit{r}}_1\hfill & {\mathit{r}}_2\hfill & {\mathit{T}}_A^c\hfill \end{array}\right]\left[\begin{array}{c}{\overline{X}}_p\hfill \\ \multicolumn{1}{c}{\begin{array}{c}{\overline{Y}}_p\hfill \\ \multicolumn{1}{c}{1}\end{array}}\end{array}\right],\hfill \end{array}$$

(10)

where ${\mathit{r}}_1,{\mathit{r}}_2,\mathrm{and}{\mathit{r}}_3$ are the columns 1, 2, and 3 of ${\mathit{R}}_A^c$ , and $\overline{w}$ is a scaling factor for image projection for the UAV camera. Let $\mathit{L}=\mathit{A}\left[\begin{array}{ccc}{\mathit{r}}_1\hfill & {\mathit{r}}_2\hfill & {\mathit{T}}_A^c\hfill \end{array}\right]$ so that Equation (10) is written in the form shown in Equation (11):

$$\begin{array}{c}\left[\begin{array}{c}{\overline{u}}_p\hfill \\ \multicolumn{1}{c}{{\overline{v}}_p}\\ \multicolumn{1}{c}{\overline{w}}\end{array}\right]=\mathit{L}\left[\begin{array}{c}{\overline{X}}_p\hfill \\ \multicolumn{1}{c}{\begin{array}{c}{\overline{Y}}_p\hfill \\ \multicolumn{1}{c}{1}\end{array}}\end{array}\right],\mathrm{where}\mathit{L}=\mathit{A}\left[\begin{array}{ccc}{\mathit{r}}_1\hfill & {\mathit{r}}_2\hfill & {\mathit{T}}_A^c\hfill \end{array}\right]\hfill \\ \left[\begin{array}{c}{\overline{u}}_p\hfill \\ \multicolumn{1}{c}{{\overline{v}}_p}\\ \multicolumn{1}{c}{\overline{w}}\end{array}\right]=\left[\begin{array}{ccc}{L}_{11}\hfill & L_{12}\hfill & L_{13}\hfill \\ \multicolumn{1}{c}{L_{21}}& L_{22}\hfill & L_{23}\hfill \\ \multicolumn{1}{c}{L_{31}}& L_{32}\hfill & L_{33}\hfill \end{array}\right]\left[\begin{array}{c}{\overline{X}}_p\hfill \\ \multicolumn{1}{c}{\begin{array}{c}{\overline{Y}}_p\hfill \\ \multicolumn{1}{c}{1}\end{array}}\end{array}\right],\hfill \end{array}$$

(11)

where $\mathit{L}$ is a homography matrix for camera projection, and $L_{\mathrm{ij}}$ are the elements of $\mathit{L}$ . By solving for $u_p$ and $v_p$ , a system of linear equations can be obtained for each point, as shown in Equation (12):

$$\begin{array}{c}{u}_p=\frac{{\overline{u}}_p}{\overline{w}}=\frac{L_{11}{\overline{X}}_{\mathrm{ap}}+L_{12}{\overline{Y}}_{\mathrm{ap}}+L_{13}}{L_{31}{\overline{X}}_{\mathrm{ap}}+L_{32}{\overline{Y}}_{\mathrm{ap}}+L_{33}},\hfill \\ v_p=\frac{{\overline{v}}_p}{\overline{w}}=\frac{L_{21}{\overline{X}}_{\mathrm{ap}}+L_{22}{\overline{Y}}_{\mathrm{ap}}+L_{23}}{L_{31}{\overline{X}}_{\mathrm{ap}}+L_{32}{\overline{Y}}_{\mathrm{ap}}+L_{33}},\hfill \end{array}$$

(12)

which can be rearranged to Equation (13):

$$\begin{array}{c}{u}_p{\overline{X}}_{\mathrm{ap}}{L}_{31}+u_p{\overline{Y}}_{\mathrm{ap}}{L}_{32}+u_p{L}_{33}-L_{11}{\overline{X}}_{\mathrm{ap}}-L_{12}{\overline{Y}}_{\mathrm{ap}}-L_{13}=0,\hfill \\ v_p{\overline{X}}_{\mathrm{ap}}{L}_{31}+v_p{L}_{32}{\overline{Y}}_{\mathrm{ap}}+v_p{L}_{33}-L_{21}{\overline{X}}_{\mathrm{ap}}-L_{22}{\overline{Y}}_{\mathrm{ap}}-L_{23}=0,\hfill \end{array}$$

(13)

which can be written in matrix form as shown in Equation (14):

$$\left[\begin{array}{ccc}{\overline{M}}_p\hfill & 0^T\hfill & -u_p{\overline{M}}_p\hfill \\ \multicolumn{1}{c}{0^T}& {\overline{M}}_p\hfill & -v_p{\overline{M}}_p\hfill \end{array}\right]·\overrightarrow{\mathit{L}}=0.$$

(14)

Here, $\overrightarrow{\mathit{L}}={[L_{11},L_{12},L_{13},L_{21},L_{22},L_{23},L_{31},L_{32},L_{33}]}^T$ . For every corner point ${\mathit{M}}_p$ , two equations are obtained; thus, four points are used to compute the values for $\overrightarrow{\mathit{L}}$ using single value decomposition (SVD). Next, the values for $\overrightarrow{\mathit{L}}$ provided by SVD are taken as an initial solution and further refined by using iterative nonlinear minimization using the Levenberg-Marquardt algorithm. ⁴⁴ This algorithm is nonlinear and provides robust pose estimation in the presence of Gaussian noise by minimizing the reprojection error, as shown in Equation (15):

$$\overrightarrow{\mathit{L}}:=\mathrm{mi}{\mathrm{n}}_{\mathrm{error}}\left(\sum _{\mathrm{iterations}}{\Vert {\overline{\mathit{m}}}_{\mathit{p}}-{\hat{\mathit{m}}}_{\mathit{p}}\Vert }^2\right)$$

(15)

where ${\hat{\mathit{m}}}_{\mathit{p}}$ is the predicted reprojection point obtained using the estimated $\overrightarrow{\mathit{L}}$ . Finally, the elements of the required extrinsic parameters $\left[{\mathit{R}}_A^c|{\mathit{T}}_A^c\right]$ of the camera are computed using Equations (16) and (17):

$$\left[\begin{array}{ccc}{\mathit{r}}_1\hfill & {\mathit{r}}_2\hfill & {\mathit{T}}_A^c\hfill \end{array}\right]={\mathit{A}}^{-1}·\mathit{L}$$

(16)

$${\mathit{r}}_3={\mathit{r}}_1\times {\mathit{r}}_2$$

(17)

${\mathit{R}}_A^c$ is first transformed to obtain the orientation of camera ${\mathit{R}}_c^A$ in O _A using the inverse operation. Since rotation matrices are orthogonal, the transpose of the rotation matrix is the inverse of that matrix, as shown in Equation (18). Similarly, the position vector of the camera with respect to ArUco is obtained using Equation (19).

$${\mathit{R}}_c^A=({\mathit{R}}_A^c{)}^T,$$

(18)

$${\mathit{T}}_c^A=-{\mathit{T}}_A^c,$$

(19)

where ${\mathit{T}}_c^A$ is the position vector of camera in O _A. Now we have the orientation of camera ${\mathit{R}}_c^A$ with respect to ${\mathit{O}}_A$ , and a series of rotation transformations is required to get the pose of the UAV in ${\mathit{O}}_g$ , as shown in Figure 12.

Figure 12.

Rotation transformations to find pose of UAV.

The series of transformations can be shown in Equation (20):

$${\mathit{R}}_d^g={\mathit{R}}_c^d·{\mathit{R}}_A^g·{\mathit{R}}_c^A,$$

(20)

where ${\mathit{R}}_A^g$ and ${\mathit{R}}_c^d$ are 3 × 3 rotation matrices that can be directly computed using the general form of rotation matrices and known angle of rotation. ⁴⁵ Here, ${\mathit{R}}_A^g$ aligns ${\mathit{O}}_A$ with ${\mathit{O}}_g$ , and ${\mathit{R}}_c^d$ aligns ${\mathit{O}}_c$ with using the known orientation of ArUco and camera (see Figures 11 and 12). Equation (21) will yield the required global position of the UAV in the pseudo map:

$${\mathit{V}}_g^d=\left[\begin{array}{c}{x}_t^m\hfill \\ \multicolumn{1}{c}{y_t^m}\\ \multicolumn{1}{c}{z_t^m}\end{array}\right]={\mathit{R}}_d^g·{\mathit{T}}_c^A+{\mathit{V}}_g^A$$

(21)

where ${\mathit{V}}_g^A={[x_{\mathrm{gA}},y_{\mathrm{gA}},z_{\mathrm{gA}}]}^T$ is the global position vector of the ArUco marker stored in the marker library in the ground station. Then, ${\theta }_t^m$ can be calculated directly from ${\mathit{R}}_d^g$ as shown in Equation (22):

$${\theta }_t^m=\mathrm{atan}2({\mathit{R}}_{d_{11}}^g,{\mathit{R}}_{d_{21}}^g).$$

(22)

Here, for $\mathrm{atan}$ , the $\mathrm{atan}2$ function included in the Math Library in Python ⁴⁶ is used to calculate the yaw angle in range $[-\pi ,\pi ]$ radians; ${\mathit{R}}_{d_{11}}^g$ is the first item in the first row, and ${\mathit{R}}_{d_{21}}^g$ is the first element of the second row of ${\mathit{R}}_d^g$ .

Crack damage segmentation using STRNet

For a damage detection task, we recently used an existing algorithm called STRNet, ¹⁸ which is an advanced crack segmentation algorithm. STRNet is a deep learning network for crack segmentation in a complex background for real-time processing (i.e., 49 FPS for a relatively large size input image (1024 × 512)) using a self-attention-based decoder, as shown in Figure 13. The input image $I_t^{m2}$ (1024 × 512) from Camera_2, as shown in Figure 1 goes through the first convolution block, which involves convolution and batch normalization ⁴⁷ with Hswish ⁴⁸ activation function. Next, the extracted feature map by the convolution block is inputted into the semantic transformer representation (STR) module, which is repeated 11 times. After that, max pooling is performed, and the result is inputted into the attention decoder that we developed. The attention decoder result after two repetitions are upsampled as the original size of the input image. All the green lines express skipped connections to keep multi-dimensional features throughout the process of STRNet. In addition, the outputs of the upsampling layer, first convolution block, and coarse upsampling block are concatenated and go through pointwise convolution (PW) to produce an output. The output provides segmented cracks at the pixel level. In the following subsections are brief descriptions of the STR module and attention decoder.

Figure 13.

STRNet architecture.

STR module

CNNs use many convolution layers to extract an object-focused feature map. The computational cost of a convolutional operator is usually dependent on kernel size (k), input channel (C), input width (W), input height (H), and output channel (O). The computational costs of a traditional (T) convolution, depthwise (DW) convolution, PW, and their combination are calculated as

$$T=k^2\mathrm{CWHO}$$

(23)

$$\mathrm{DW}=k^2\mathrm{CWH}$$

(24)

$$\mathrm{PW}=\mathrm{CWHO}$$

(25)

$$\mathrm{DW}+\mathrm{PW}=\mathrm{CHW}(k^2+O)$$

(26)

As shown in Equations (23) to (26), the cost is heavily affected by C, H, and W. To reduce computational costs, a pooling layer or convolution with two strides is usually applied. When we increase the size of the stride, W and H from Equations (23) and (26) are reduced. Therefore, proper sizes of the strides in each layer and operator are important to reduce the computational cost of a specific network, but the increase of stride may cause the loss of important features. This is a very critical problem in crack segmentation, since crack lines are thin, and specific feature loss may critically reduce the performance of the segmentation. To overcome the issue, an STR module was developed. The STR module consists of “Compress,”“Attention,” and “Restoration” modules. The base module is mainly composed of the PW and DW convolution blocks. As shown in Figure 13 Compress module applies DWs to increase stride sizes, and the compressed feature map is passed to the Attention module which includes a squeeze and excitation block (SE block). ⁴⁹

In the SE block, to produce the attention map, global average pooling is applied for the input feature map, and two linear functions and all activation functions, such as rectified linear unit ⁵⁰ and H-sigmoid, ⁵¹ are performed as a squeezing process. Then, the squeezed feature map is expanded by coping of the squeezed map to the input size of the SE block to apply the matrix multiplication (■). Finally, it is multiplied by the input of the SE block. The compress and restoration block (CRB) compresses the width and height of the input feature map and recovers the compressed size of the feature map to the original input feature map of the STR module by combining the reserved features using the skipped connection. The concatenation operation helps prevent a vanishing gradient and the loss of important features.

Attention decoder

After the STR encoder, we apply the attention decoder, as shown in Figure 14. The original concept of attention comes from a previous study. ⁵² Recently, an improved attention-based model, transformer, proposed by Vaswani et al., ⁵³ applies multi-head attention using Key, Query, and Value. Query, Key, and Value are calculated using scaled dot-product to find the attention map and enhance the important features. This can be expressed by Equation (27)

$$\mathrm{Attention}=\mathrm{Softmax}\left(\frac{\mathrm{Quer}{\mathrm{y}}^T*\mathrm{Key}}{\sqrt{D}}\right)*\mathrm{Valu}{\mathrm{e}}^T$$

(27)

Figure 14.

Attention decoder for STRNet.

Data generation for training of STRNet

Deep learning-based semantic segmentation methods have been actively adopted in pixel-level crack segmentation research in the last few years. However, annotating highly accurate segmentation data entails high labor costs because of labeling ground truth. For example, the annotation at the pixel level for PASCAL VOC is evaluated to be 239.7 seconds per image. ⁵⁴ Annotating crack images take approximately 20–40 min. ⁵⁵ To reduce the time required to prepare the dataset, we considered two options: (1) a public dataset and (2) raw images only.

Public concrete crack datasets exist.^10,56–58 Although Choi and Cha ¹⁰ created a dataset using large-sized images, most other public crack datasets contain small-sized images, and the image and mask occasionally have errors. Furthermore, almost all public crack datasets have images with very simple backgrounds. For this reason, many public crack images were not selected for our training dataset, and a new dataset was developed ¹⁸ (see Table 1).

Table 1.

Crack datasets for training and testing (Kang and Cha, 2021).

Dataset	Train	Test	Total
Crack 1748 (1024 × 512)	1203	545	1748
UAV 54 (1024 × 512)	—	54	54
Total	1203 (12,030)	599	1802

UAV: unmanned aerial vehicle.

STRNet training and evaluation

In this experiment, STRNet was trained using the data tabulated in Table 1. The Python programming language ⁴⁶ with the Pytorch 1.6 deep learning library ⁵⁹ was applied to implement STRNet. It was trained with four Titan XP graphics processing units (GPUs). The workstation had an Intel Core i7-6850 K central processing unit (CPU) and 128 GB memory. To train our models, the Nvidia Apex distributed data parallel training library was used to utilize the multi-GPUs. The input image size was 1024 × 512, which is randomly cropped if the image size is greater than the input size. The use of proper loss function is critical; therefore, we investigated several loss functions, such as cross-entropy loss, dice cross-entropy loss, and mIoU. Eventually, the focal Tversky loss function showed the best performance. ⁶⁰ The formulation of the loss function is presented in Equation (28)

$$\mathrm{Focal}-\mathrm{Tversky}\mathrm{loss}={(1-\frac{\mathrm{TP}+\epsilon }{\mathrm{TP}+\theta *\mathrm{FP}+\mu *\mathrm{FN}+\epsilon })}^{\tau }$$

(28)

The hyperparameters $\epsilon$ , $\theta$ , $\mu$ , and $\tau$ are defined as 0.5, 0.5, 1.0, and 1.3, respectively. $\theta$ scales FP (false positives), while $\mu$ scales FN (false negatives). $\epsilon$ is a small positive number for smoothing of the training process. $\tau$ increases focal loss. These values were investigated by trial and error.

An Adam optimizer was applied as the optimizer, and the hyperparameters of ${\beta }_1$ and ${\beta }_2$ were set as 0.9 and 0.999, respectively. ⁶¹ To validate the superior performance of STRNet, algorithms developed in previous research were also tested for comparison, such as Attention U-Net, MobileNetV3 Large, ⁵¹ and Deeplab v3+. ⁶² We used all the same training data and testing data for a fair comparison. The evaluation results of the four networks are represented in Figure 15.

Figure 15.

Example results of the comparative studies on complicated images: (a) concrete crack with a building in background, (b) concrete crack on pavement, (c) thin crack on concrete with grass in background and (d) crack with slab joints.

To calculate mIoU, we used Equation (29) since it can properly consider FP and FN compared to the other recall, precision, and F1 scores

$$\mathrm{mIoU}=\mathrm{mean}\left(\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FP}+\mathrm{FN}}\right)$$

(29)

Using the dataset presented in Table 1, we conducted extensive training and testing, and the results are presented in Table 2. STRNet achieved the best $\mathrm{mIoU}$ (92.5%) and F1 score (92.08%) from 545 various testing images having complex scenes and the best $\mathrm{mIoU}$ (91.8%) and F1 score (90.7%) from 54 images from the UAV camera. Some of the examples are illustrated in Figure 15.

Table 2.

Performance comparison of different models for crack detection on UAV images and crack segmentation on ground images.

Model	Crack 1748 (545)		UAV 54		Speed
	F1 Score	mIoU	F1 Score	mIoU	V100 (FPS)
MobileNetV3	85.61	87.1	74.4	80.0	71
Attention U-Net	88.3	89.1	80.6	84	17
Deeplab v3+	80.36	83.24	69.2	76.9	30.2
STRNet	92.08	92.5	90.7	91.8	47.8

FPS: frames per second; mIoU: mean intersection over union; UAV: unmanned aerial vehicle.

The bold values are highlighting mIoU performance of networks.

MobileNetV3 included a false positive object in images a, c, and d. Deeplab v3+ showed more severe false positive detection in all the images. Attention U-Net detected expansion joints as cracks in images a, b, and c. The difficulty of the crack 1748 dataset lies in complex scenes and various shapes of crack-like objects. STRNet successfully overcame most of these challenging problems and showed very accurate segmentation results. Based on these achievements, STRNet was implemented on the images taken from the UAV camera for inspection. The results are presented in Section “Case studies.”

Case studies

Parrot Bebop 2 Power Drone ⁶³ and Anafi ⁶⁴ (https://www.parrot.com/en/about-parrot) were selected to test autonomous UAVs, as shown in Figure 16. This allows the developer to develop their own controller for autonomous navigation. A laptop computer with an Intel Core i7 2.6 GHz CPU and Nvidia Geforce (https://www.nvidia.com/en-us/) 1060 GPU communicated with the UAV through Wi-Fi-based communication. The front camera of the UAV was used for obstacle detection, and due to the limitation of the payload, a lightweight camera was chosen as the second camera for damage (crack) segmentation. As the second camera, we used the Hawkeye Firefly Micro Cam 160 model, as indicated in the red box in Figure 16. We modified the hardware of the camera to install it on the UAV. Two UAV localization methods, Marvelmind USB for indoor experiments and ArUco marker-based localization for outdoor experiments, were used, as explained in Figure 1 in Section “Methodology.” The details of experimental testing and validation for indoor and outdoor environments are explained in the following subsections “Indoor experiments” and “Outdoor experiments,” respectively.

Figure 16.

Hardware settings: (a) USB-based approach with Marvelmind beacon; inspection camera: Firefly Micro Cam 160 model, Bebop Power 2, and; (b) ArUco marker-based approach using an Anafi UAV.

Indoor experiments

The proposed OAM was initially tested in an indoor environment with a USB-based localization method. In order to investigate the performance, it was compared with existing OAMs such as the MonodepthV2 (Godard et al., 2019) ³² and AdaBins (Bhat et al., 2020) ³³ approaches. Details of the OAM and STRNet experimental tests are provided below.

Obstacle avoidance

In order to conduct experimental tests to validate the proposed obstacle avoidance algorithm and crack damage detection, we generated a planned trajectory for autonomous navigation of the UAV in the laptop ground station by assigning a series of waypoints (expressed as blue lines, as shown in Figure 17). The distance between each waypoint was set as 50 cm. Two and three groups of obstacles were set, which are illustrated as black rectangular boxes. The locations of detected cracks in the concrete floor are illustrated as red triangles. Specific waypoints were expressed as red circles in obstacle scenario 1. As shown in Figure 17, two cases of obstacle scenarios were well avoided by using our proposed OAM. The orange lines express the trajectory to avoid obstacles. Also, during the navigations, all three cracks in the concrete floor were detected well. All the dimensions are expressed as cm units.

Figure 17.

Flight path: (a) obstacle avoidance with two clusters of obstacles, and (b) obstacle avoidance with three clusters of obstacles.

There were three chairs in each group of obstacles, and the proposed OAM (see Section “Obstacle clustering”) operated well during the autonomous navigation. As shown in Figure 18, YOLOv3 detected obstacles successfully, and the K-means clustering algorithm created a group of boxes based on features and converted them into a single obstacle box at waypoints 1, 5, and 11.

Figure 18.

OAM performances based on YOLOv3 and K-means clustering at waypoints 1, 5, and 11.

As comparative study, our approach was compared to existing deep learning-based monocular depth estimation algorithms such as MonodepthV2 (Godard et al., 2019) ³² and AdaBins (Bhat et al., 2020). ³³ Using the two tests, 20 image frames with obstacles in the scenes from each autonomous navigation were selected to compare the performance of the obstacle identification/clustering. A total of 40 image frames were used as the input for the three methods and tested, as shown in Figure 19.

Figure 19.

Detected obstacles in three methods: (a) Farther away obstacles in center, (b) Closer obstacles, (c) Multiple obstacle groups, (d) Farther away obstacles to right, (e) Partially visible obstacle, (f) Two closer obstacles, one partially visible, (g) Two closer obstacles, one partially visible to left, and (h) Three obstacles fully visible to right grouped.

Our object clustering algorithm and AdaBins worked well to identify obstacles in various scenarios in terms of location within the image frame. However, MonodepthV2 failed to identify the chairs as shown in Figure 19(a) to (d). Furthermore, empty spaces were detected as obstacles consistently in Figure 19(a) to (h).

While MonodepthV2 failed to provide the proper quality of the depth map to identify obstacles, AdaBins and Ours achieved very successful results. To evaluate the performances, we use the three most common evaluation metrics: precision, recall, and F1 score, as expressed in Equations (30)–(32). True positive, true negative, false positive, and false negative cases are abbreviated to TP, TN, FP, and FN, respectively.

$$\mathrm{Precision}=\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FP}}$$

(30)

$$\mathrm{Recall}=\left(\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FN}}\right)$$

(31)

$$F1\mathrm{score}=2\times \frac{\mathrm{Precision}\times \mathrm{Recall}}{\mathrm{Precision}+\mathrm{Recall}}$$

(32)

All the calculated evaluation metrics are tabulated in Table 3. The results of 40 images were manually checked. There were two obstacles in scenario 1 and three obstacles in scenario 2, as shown in Figure 17, which appeared in the 40 image frames 95 times. AdaBins and Ours showed satisfactory results in these two cases of autonomous navigation with different obstacle scenarios, as presented in Table 3.

Table 3.

Evaluation result of obstacle state for each algorithm

Algorithm	Precision	Recall	F1_score	FPS
AdaBins	1.00	1.00	1.00	0.3
MonodepthV2	0.76	0.78	0.77	0.5
Ours	1.00	1.00	1.00	10

FPS: frames per second.

However, the processing speed of the AadBins was just 0.3 FPS, which may delay the overall inspection process due to the low speed of the OAM, since videos are commonly 30 FPS. However, our algorithm was still 10 FPS using an old 1060 GPU. Also, as shown in Figure 19(a), (b), and (d), AdaBins could not detect the legs of the chairs. It can be potentially dangerous in infrastructure inspections if small, thin obstacles are not properly detected during navigation for monitoring.

STRNet results for indoor case

Figure 20 shows the three different cracks presented in Figure 17 as red triangles (a, b, c). The cracks in the images from the UAV camera are seen as very unclear and blurry due to the vibration of the UAV. The results of the Deeplab v3+ and MobileNet v3 suffered from loss of the crack pixels throughout the line of the cracks (Figure 20(a) and (c)). Attention U-Net showed good results in image c, but it missed the end of the crack in images a and b. Among these algorithms, STRNet showed very good performance in all images, with 91.8% mIoU.

Figure 20.

Comparison between MobileNetV3, Attention U-Net, Deeplab v3+, and STRNet: (a) crack image 1, (b) crack image 2, and (c) crack image 3.

Outdoor experiments

After obtaining successful results from indoor experiments, outdoor experiments were performed in a multi-level parking structure in the Fort Garry campus of the University of Manitoba. The top floor was used to conduct the autonomous UAV experiments. The wind direction was along the southeast, with wind gusts of more than 30 km/h.

The average height of the UAV’s planned waypoint was 7 m from ground level. This natural, real-world environment was used to validate the robustness of the developed systems. ArUco markers were installed at every 1 m on the ceiling in the designated section, as shown in Figure 21. For safety, nets were installed instead of windows. In the experimental setup, the UAV was under the impact of strong to medium wind gusts. Different sets of planned waypoints were used to confirm the validity of the proposed method of UAV control via ArUco markers. The overall outdoor experiment area is shown in Figure 21. During the experiments in the parkade garage, the lighting conditions were challenging due to the low light levels and the presence of occluding lights. These conditions can affect the performance of a camera-based system, as the low light levels can make it difficult for the camera to capture clear images, and the occluding lights can create glare and shadows that may obscure objects of interest. These markers are designed to be easily detected by a camera system, even when the lighting conditions are poor.

Figure 21.

Outdoor experimental setup using the ArUco-based navigation system with chairs as obstacles.

Yaw control of UAV

The most important feature of using computer vision-based ArUco marker-based localization is the robust estimation of yaw ${\theta }_t^m$ , which is critical to follow a set of planned waypoints $(x_t^p,y_t^p,z_t^p,\mathrm{and}{\theta }_t^p)$ . In our outdoor experiments, tests were conducted to show the robustness of yaw control using ArUco markers under a GPS-denied and high magnetic interference environment in a parking structure. The UAV was given the task of moving in a straight line for 10 meters while maintaining an angle of 0° along the planned waypoints shown in Figure 22(a).

Figure 22.

Comparing the performance of (a) deviation in yaw control and (b) deviation in path control for the ArUco marker and USB-based localization methods in an outdoor parking structure.

It was observed that the USB sensors resulted in poor readings when close to steel girders due to magnetic interference. The noisy and unreliable readings from the USB resulted in poor control of the UAV. As a result, the UAV deviated from the planned waypoints when relying on USB data, as shown in Figure 22(b). The yaw deviation $({\theta }_t^m-{\theta }_t^p)$ for USB was in the range of $[-25°\mathrm{to}31°]$ with an SD of 16.43° and a root mean square deviation (RMSD) of 18.14°. Similarly, the RMSD value for the flight path while using USB was 0.48 m.

On the other hand, the ArUco marker-based localization was not impacted by magnetic interference and provided the robust yaw values required to control the UAV under the same challenging and complex environment. The deviation $({\theta }_t^m-{\theta }_t^p)$ in using ArUco markers was limited to within $[-{13}^{°},{10}^{°}]$ with an SD of 4.974° and an RMSD value of 7.89°. The RMSD value for path following with ArUco markers was 0.114 m. Thus, in the outdoor environment, the ArUco marker-based localization decreased the error in yaw control by 60.45% and decreased the error in path following by 67.29%.

Obstacle avoidance

Based on the comparative studies of the performances of the USB-based and ArUco marker-based localization methods, we chose ArUco marker-based localization for outdoor experimental validation of our proposed OAM.

To avoid obstacles such as chairs, the UAV camera must be able to capture images or video frames from the front side. For this, the camera feed $I_t^{m1}$ is used to detect obstacles using YOLOv3, as discussed in Section “Obstacle avoidance method.” Two sets of obstacle configurations were used to verify the performance of the OAM algorithm. In this experiment, the planned route was a $3.1\mathrm{m}\times 15\mathrm{m}$ rectangular path for the collection of data from the ceiling of parkade structure. Sets of obstacles were placed at 4 m and at 9 m along the y-axis, as shown in Figure 23. The resulting path followed by the UAV is shown in Figure 23 as an orange-colored line. Both obstacles were successfully avoided by the autonomous UAV. The red triangle markers of A and B in Figure 23 are the locations of the detected concrete slap cracks analyzed by STRNet, which is discussed below. Moreover, as shown in Figure 24, the OAM was able to successfully detect and avoid one and two chairs placed in the planned way point of the UAV using the real-time object detection algorithm. These results show that the UAV can successfully navigate and avoid obstacles, regardless of the number of chairs present, and demonstrates the robustness and flexibility of our UAV’s obstacle avoidance capabilities.

Figure 23.

OAM path versus planned path when navigating through: (a) one group of obstacles, and (b) two groups of obstacles.

Figure 24.

Proposed OAM performance for one and two chairs.

STRNet results for outdoor case

In Figure 25, the performance of STRNet on the data collected from the outdoor parkade is shown. Three images of two separate crack images are presented in Figure 25; their positions are shown as red triangles in Figure 23. Similar to the indoor experiments, the Deeplab v3 and MobileNetV3 performances were degraded as compared to that of Attention U-Net (Figure 25(b) and (c)). Overall, STRNet performed consistently well over all testing datasets and outperformed all the models in comparison. The crack images of Figure 25(a) are from the area indicated by red triangle A in Figure 23, and those in Figure 25(b) and (c) from that of red triangle B in Figure 23.

Figure 25.

Comparison of crack detection methods using MobileNetV3, Attention UNet, Deeplab v3+, and STRNet on UAV captured images of parkade ceiling: (a) Crack 1, (b) Crack 2, and (c) Crack 3 are highlighted for evaluation during autonomous inspection.

Conclusion and future direction

To develop a reliable autonomous UAV system for SHM, we developed a framework for obstacle-avoiding autonomous flight methods through fiducial marker-based UAV localization. The contributions of the paper are as follows:

1. A complex autonomous UAV system for SHM was developed by careful integration of a new OAM, a new localization method, and a state-of-the-art real-time crack segmentation method.

2. A new OAM was developed with YOLOv3 (as an example network, but it can be replaced by any recent version of YOLO) and an obstacle clustering method based on the K-mean clustering method.

3. The developed OAM was compared to existing methods, and it outperformed with a higher FPS and robust obstacle detection compared to those of the traditional OAM methods.

4. A new autonomous UAV localization method was introduced for autonomous UAVs for SHM with fiducial ArUco markers. Due to the nature of the ArUco marker, it does not require regular battery charging and is not vulnerable to environmental electromagnetic fields.

5. The ArUco marker-based method decreased localization error by 67.29% and yaw control error by 60.45% compared to those of the USB method, respectively.

6. The new OAM, fiducial ArUco marker-based autonomous UAV system, was implemented in indoor and outdoor environments as GPS-denied areas.

7. The developed autonomous UAV system including a new OAM and localization method was integrated with a state-of-the-art crack segmentation network (STRNet).

8. STRNet ¹⁸ showed superior performance (mIoU 92.5%) compared to recently developed deep CNNs (mIoU 89.1%) for parking structure crack segmentation.

In future work, we plan to integrate new hybrid techniques, such as 3D sensor fusion, to estimate the distance of unknown obstacles for autonomous UAVs and integrate the developed method with a digital twin for real-time damage mapping and management.