Unmanned aerial systems (UASs) are increasingly applied for bridge inspection. A vision-guided UAS with a lightweight convolutional neural network is developed to detect and locate bridge cracks, spalling, and corrosion. The contributions are as follows: (1) To address the problem that traditional UASs are global positioning system (GPS) required while GPS signals under bridge bottom generally are weak. A vision-guided UAS is designed and applied, in which a stereo vision-inertial fusion method is used to provide position data instead of GPS and an ultrasonic ranger is applied to avoid obstacles. (2) Most of the deep learning-based damage detection methods are offline detection, which is unsuitable for UAS-based inspection because the endurance time is limited. To solve this problem, a lightweight end-to-end object detection network is proposed, by replacing the backbone of the original You Only Look Once v3 network with MobileNetv2, and the proposed network of much faster inference speed can be transplanted to the onboard computer of the designed UAS so that real-time edge computing can be performed during inspection. (3) A damage location method based on vision positioning data and simultaneous localization and mapping is also proposed to meet the urgent needs of locating damage in the whole structure. Finally, the proposed system is applied to inspect a long-span bridge to detect and locate the most common damages: crack, spalling, and corrosion with high accuracy and efficiency, which verified the practicability of the system.
ChangPCFlatauALiuSC. Review paper: health monitoring of civil infrastructure. Struct Health Monitoring2003; 2(3): 257–267.
2.
YoonHShinJSpencerBFJr. Structural displacement measurement using an unmanned aerial system. Computer-Aided Civil Infrastructure Eng2018; 33(3): 183–192.
3.
JiangSZhangJ. Real‐time crack assessment using deep neural networks with wall‐climbing unmanned aerial system. Computer-Aided Civil Infrastructure Eng2020; 35(6): 549–564.
4.
TianYZhangCJiangS, et al.Noncontact cable force estimation with unmanned aerial vehicle and computer vision. Computer-Aided Civil Infrastructure Eng2020; 36: 73–88. DOI: 10.1111/mice.12567
5.
DorafshanSThomasRJMaguireM. Fatigue crack detection using unmanned aerial systems in fracture critical inspection of steel bridges. J Bridge Engineering2018; 23(10): 04018078.
6.
KarimMMQinRChenG, et al.A semi-supervised self-training method to develop assistive intelligence for segmenting multiclass bridge elements from inspection videos. Struct Health Monit2021; 21: 835–852.
7.
KangDChaY-J. Autonomous UAVs for structural health monitoring using deep learning and an ultrasonic beacon system with geo-tagging. Computer-Aided Civil Infrastructure Eng2018; 33(10): 885–902.
8.
NguyenTMZainiAHGuoK, et al.An ultra-wideband-based multi-UAV localization system in GPS-denied environments.In: 2016 International Micro Air Vehicles Conference. 2016.
9.
TiemannJWietfeldC. Scalable and precise multi-UAV indoor navigation using TDOA-based UWB localization. In: 2017 International Conference on Indoor Positioning and Indoor Navigation (IPIN). IEEE, 2017.
10.
MasieroAFissoreFAntonelloR, et al.A comparison of UWB and motion capture UAV indoor positioning. International Archives of the Photogrammetry. Remote Sensing Spat Inf Sci, 2019.
11.
QinTLiPShenS. Vins-mono: a robust and versatile monocular visual-inertial state estimator. IEEE Trans Robotics2018; 34(4): 1004–1020.
12.
SmolyanskiyNKamenevASmithJ, et al.Toward low-flying autonomous MAV trail navigation using deep neural networks for environmental awareness. In: 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2017.
13.
FlorencePRCarterJWareJ, et al.Nanomap: Fast, uncertainty-aware proximity queries with lazy search over local 3d data. In: 2018 IEEE International Conference on Robotics and Automation (ICRA)IEEE, 2018.
14.
SantosMCPSantanaLVMartinsMM, et al.Estimating and controlling uav position using rgb-d/imu data fusion with decentralized information/Kalman filter. In: 2015 IEEE international conference on industrial technology (ICIT). IEEE, 2015.
15.
QinTPanJCaoS, et al.A general optimization-based framework for local odometry estimation with multiple sensors. arXiv preprint arXiv:1901.03638, 2019.
16.
MillerIDCladeraFCowleyA, et al.Mine tunnel exploration using multiple quadrupedal robots. IEEE Robotics Automation Lett2020; 5(2): 2840–2847.
17.
GrabeVBülthoffHHGiordanoPR. Robust optical-flow based self-motion estimation for a quadrotor UAV. In: 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems. IEEE, 2012.
18.
González-deSantosLMMartínez-SánchezJGonzález-JorgeH, et al.UAV payload with collision mitigation for contact inspection. Automation in Construction2020; 115: 103200.
19.
SpencerBFHoskereVNarazakiY. Advances in computer vision-based civil infrastructure inspection and monitoring. Engineering2019; 5(2): 199–222.
20.
BaoYChenZWeiS, et al.The state of the art of data science and engineering in structural health monitoring. Engineering2019; 5(2): 234–242.
21.
YangXWeiSBaoY, et al.Automatic seismic damage identification of reinforced concrete columns from images by a region-based deep convolutional neural network. Struct Control Health Monit2019; 26: e2313.
22.
ChaY-JChoiWBüyüköztürkO. Deep learning-based crack damage detection using convolutional neural networks. Computer-Aided Civil Infrastructure Eng2017; 32(5): 361–378.
23.
ZhangAWangKCPLiB, et al.Automated pixel-level pavement crack detection on 3D asphalt surfaces using a deep-learning network. Computer-Aided Civil Infrastructure Eng2017; 32(10): 805–819.
24.
JangKKimNAnYK. Deep learning-based autonomous concrete crack evaluation through hybrid image scanning. Struct Health Monit2019; 18(5–6): 1722–1737.
25.
NiFZhangJChenZ. Pixel-level crack delineation in images with convolutional feature fusion. Struct Control Health Monit2019; 26(1): e2286.
26.
DengJLuYLeeVCS. Concrete crack detection with handwriting script interferences using faster region‐based convolutional neural network. Computer-Aided Civil Infrastructure Eng2020; 35(4): 373–388.
27.
XuYBaoYChenJ, et al.Surface fatigue crack identification in steel box girder of bridges by a deep fusion convolutional neural network based on consumer-grade camera images. Struct Health Monit2019; 18(3): 653–674.
28.
DengJLuYLeeVCS. Imaging-based crack detection on concrete surfaces using You Only Look Once network. Struct Health Monit2020; 20: 448–499.
29.
MaedaHSekimotoYSetoT, et al.Road damage detection and classification using deep neural networks with smartphone images. Computer-Aided Civil Infrastructure Eng2018; 33(12): 1127–1141.
30.
KimHAhnEShinM, et al.Crack and noncrack classification from concrete surface images using machine learning. Struct Health Monit2019; 18(3): 725–738.
31.
ZhangYSunXLohKJ, et al.Autonomous bolt loosening detection using deep learning. Struct Health Monit2020; 19(1): 105–122.
32.
AthaDJJahanshahiMR. Evaluation of deep learning approaches based on convolutional neural networks for corrosion detection. Struct Health Monit2018; 17(5): 1110–1128.
33.
ChaY-JChoiWSuhG, et al.Autonomous structural visual inspection using region-based deep learning for detecting multiple damage types. Computer-Aided Civil Infrastructure Eng2018; 33(9): 731–747.
34.
HoskereVNarazakiYHoangTA., et al.MaDnet: multi-task semantic segmentation of multiple types of structural materials and damage in images of civil infrastructure. J Civil Struct Health Monit2020; 10: 757–773.
35.
XuYBaoYZhangY, et al.Attribute-based structural damage identification by few-shot meta learning with inter-class knowledge transfer. Struct Health Monit2020. DOI: 10.1177/1475921720921135
36.
BaoYLiH. Machine learning paradigm for structural health monitoring. Struct Health Monit2020: 147592172097241. DOI: 10.1177/1475921720972416
37.
QuyTBKimJ-M. Crack detection and localization in a fluid pipeline based on acoustic emission signals. Mech Syst Signal Process2021; 150: 107254.
38.
LeHVKimDJ. Detecting crack and damage location in self-sensing fiber reinforced cementitious composites. Construction Building Mater2020; 240: 117973.
39.
GuanRLuYDuanW, et al.Guided waves for damage identification in pipeline structures: A review. Struct Control Health Monit2017; 24(11): e2007.
40.
LiuYFNieXFanJS, et al.Image‐based crack assessment of bridge piers using unmanned aerial vehicles and three‐dimensional scene reconstruction. Computer-Aided Civil Infrastructure Eng2020; 35(5): 511–529.
41.
YeumCMChoiJDykeSJ. Automated region-of-interest localization and classification for vision-based visual assessment of civil infrastructure. Struct Health Monit2019; 18(3): 675–689.
42.
YoonSGwonGHLeeJH, et al.Three-dimensional image coordinate-based missing region of interest area detection and damage localization for bridge visual inspection using unmanned aerial vehicles. Struct Health Monit2020: 1475921720918675.
43.
ChoiJYeumCDykeS, et al.Computer-aided approach for rapid post-event visual evaluation of a building façade. Sensors2018; 18(9): 3017.
44.
AttardLDebonoCJValentinoG, et al.Tunnel inspection using photogrammetric techniques and image processing: A review. ISPRS J Photogrammetry Remote Sensing2018; 144: 180–188.
45.
WonJParkJWShimC, et al.Bridge-surface panoramic-image generation for automated bridge-inspection using deepmatching. Struct Health Monit2020: 1475921720930380.
46.
DavisonAJReidIDMoltonND, et al.MonoSLAM: real-time single camera SLAM. IEEE Transactions Pattern Analysis Machine Intelligence2007; 29(6): 1052–1067.
47.
ForsterCPizzoliMScaramuzzaD. SVO: Fast semi-direct monocular visual odometry. In: 2014 IEEE international conference on robotics and automation (ICRA). IEEE, 2014.
48.
EngelJSchöpsTCremersD. LSD-SLAM: Large-scale direct monocular SLAM. In: European conference on computer vision. Cham: Springer, 2014.
Mur-ArtalRTardósJD. Orb-slam2: an open-source slam system for monocular, stereo, and rgb-d cameras. IEEE Trans Robotics2017; 33(5): 1255–1262.
51.
BloeschMOmariSHutterM, et al.Robust visual inertial odometry using a direct EKF-based approach. In: 2015 IEEE/RSJ international conference on intelligent robots and systems (IROS). IEEE, 2015.
52.
RedmonJDivvalaSGirshickR, et al.You only look once: unified, real-time object detection. Proc IEEE Conference Computer Vision Pattern Recognition2016: 779–788.
53.
RedmonJFarhadiA. Yolov3: an incremental improvement. arXiv preprint arXiv:1804.02767, 2018.
54.
LiuWAnguelovDErhanD, et al.Ssd: Single shot multibox detector. In: European conference on computer vision. Cham: Springer, 2016.
55.
HowardAGZhuMChenB, et al.Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861, 2017.
56.
SandlerMHowardAZhuM, et al.Mobilenetv2: inverted residuals and linear bottlenecks. Proc IEEE Conference Computer Vision Pattern Recognition2018: 4510–4520.
