Bolt loosening detection plays an important role in structure maintenance, but traditional detection methods prove inadequate for large-scale infrastructure inspection. Recent advances in computer vision and deep learning (DL) provide more accurate and rapid detection capabilities, but the inherent issue of the model’s black-box nature cannot be ignored. This article proposes an explainable bolt loosening detection framework, named X-Bolt. This framework features a novel visual target establishing physical–visual correlations with bolt states, coupled with BoltYOLO, a customized neural network achieving accurate and real-time detection. Through gradient-weighted class activation mapping-based multiperspective interpretation, the alignment between the model’s decision-making rationale and the visual target is made explicit, and the feature learning procedure of the model, particularly the attention allocation behavior, is effectively visualized. In summary, X-Bolt provides an efficient and interpretable solution for vision-based bolt loosening detection, while providing a generalizable framework for similar inspection tasks involving DL, thereby enhancing trustworthiness and deployability in artificial intelligence-based structural health monitoring.
HaoGJianhuaLHuihuaF.Review on anti-loosening methods for threaded fasteners. Chin J Aeronaut2022; 35(2): 47–61.
2.
NikraveshSMYGoudarziM. A review paper on looseness detection methods in bolted structures. Lat Am J Solids Struct2017; 14(12): 2153–2176.
3.
HuangJLiuJGongH, et al. A comprehensive review of loosening detection methods for threaded fasteners. Mech Syst Signal Process2022; 168: 108652.
4.
VandEHOskoueiRChakherlouT.An experimental method for measuring clamping force in bolted connections and effect of bolt threads lubrication on its value. World Acad Sci Eng Technol2008; 22: 457–460.
5.
WangTTanBLuG, et al. Bolt pretightening force measurement based on strain distribution of bolt head surface. J Aerosp Eng2020; 33(4): 04020034.
6.
WangTSongGWangZ, et al. Proof-of-concept study of monitoring bolt connection status using a piezoelectric based active sensing method. Smart Mater Struct2013; 22(8): 087001.
7.
NguyenTTTaQBHoDD, et al. A method for automated bolt-loosening monitoring and assessment using impedance technique and deep learning. Dev Built Environ2023; 14: 100122.
8.
TanBWangTFangQ, et al. Bolt looseness monitoring using dynamic mode decomposition with piezoelectric active sensing. Measurement2024; 238: 115204.
9.
ZhouLChenSXNiYQ, et al. Emi-GCN: a hybrid model for real-time monitoring of multiple bolt looseness using electromechanical impedance and graph convolutional networks. Smart Mater Struct2021; 30(3): 035032.
10.
TaoWShaopengLJunhuaS, et al. Health monitoring of bolted joints using the time reversal method and piezoelectric transducers. Smart Mater Struct2016; 25(2): 025010.
11.
YangJChangFK. Detection of bolt loosening in c–c composite thermal protection panels: I. diagnosticprinciple. Smart Mater Struct2006; 15(2): 581.
12.
DuFXuCWuG, et al. Preload monitoring of bolted l-shaped lap joints using virtual time reversal method. Sensors2018; 18(6): 1928.
13.
ZhangMShenYXiaoL, et al. Application of subharmonic resonance for the detection of bolted joint looseness. Nonlinear Dyn2017; 88: 1643–1653.
14.
WangMAoWKBownjohnJ, et al. A novel gradient-based matching via voting technique for vision-based structural displacement measurement. Mech Syst Signal Process2022; 171: 108951.
15.
WangMAoWKBownjohnJ, et al. Completely non-contact modal testing of full-scale bridge in challenging conditions using vision sensing systems. Eng Struct2022; 272: 114994.
16.
ParkJHHuynhTCChoiSH, et al. Vision-based technique for bolt-loosening detection in wind turbine tower. Wind Struct2015; 21(6): 709–726.
17.
NguyenTCHuynhTCRyuJY, et al. Bolt-loosening identification of bolt connections by vision image-based technique. In: YuTGyekenyesiALShullPJ, et al. (eds) Nondestructive characterization and monitoring of advanced materials, aerospace, and civil infrastructure, vol. 9804. Bellingham WA: SPIE, 2016, pp. 227–243.
18.
ChaYJYouKChoiW.Vision-based detection of loosened bolts using the Hough transform and support vector machines. Autom Constr2016; 71: 181–188.
19.
RamanaLChoiWChaYJ.Fully automated vision-based loosened bolt detection using the viola–jones algorithm. Struct Health Monit2019; 18(2): 422–434.
20.
VoulodimosADoulamisNDoulamisA, et al. Deep learning for computer vision: a brief review. Comput Intell Neurosci2018; 2018(1): 7068349.
21.
TongKWuYZhouF.Recent advances in small object detection based on deep learning: a review. Image Vis Comput2020; 97: 103910.
DorafshanSThomasRJMaguireM.Comparison of deep convolutional neural networks and edge detectors for image-based crack detection in concrete. Constr Build Mater2018; 186: 1031–1045.
24.
YeumCMDykeSJRamirezJ.Visual data classification in post-event building reconnaissance. Eng Struct2018; 155: 16–24.
25.
VetrivelAGerkeMKerleN, et al. Disaster damage detection through synergistic use of deep learning and 3D point cloud features derived from very high resolution oblique aerial images, and multiple-kernel-learning. ISPRS J Photogramm Remote Sens2018; 140: 45–59.
26.
ChaYJChoiWSuhG, et al. Autonomous structural visual inspection using region-based deep learning for detecting multiple damage types. Comput Aided Civ Infrastruct Eng2018; 33(9): 731–747.
27.
GaoYMosalamKM.Deep transfer learning for image-based structural damage recognition. Comput Aided Civ Infrastruct Eng2018; 33(9): 748–768.
28.
AliRKangDSuhG, et al. Real-time multiple damage mapping using autonomous UAV and deep faster region-based neural networks for GPS-denied structures. Autom Constr2021; 130: 103831.
29.
KumarDDFangCZhengY, et al. Semi-supervised transfer learning-based automatic weld defect detection and visual inspection. Eng Struct2023; 292: 116580.
30.
ChaYJAliRLewisJ, et al. Deep learning-based structural health monitoring. Autom Constr2024; 161: 105328.
31.
HuynhTCParkJHJungHJ, et al. Quasi-autonomous bolt-loosening detection method using vision-based deep learning and image processing. Autom Constr2019; 105: 102844.
32.
ZhangYSunXLohKJ, et al. Autonomous bolt loosening detection using deep learning. Struct Health Monit2020; 19(1): 105–122.
33.
ZhaoXZhangYWangN.Bolt loosening angle detection technology using deep learning. Struct Control Health Monit2019; 26(1): e2292.
34.
YangXGaoYFangC, et al. Deep learning-based bolt loosening detection for wind turbine towers. Struct Control Health Monit2022; 29(6): e2943.
35.
RazzakMNazSZaibA, et al. Classification in bioapps. Cham: Springer, 2018, pp. 323–350.
36.
KimJCannyJ. Interpretable learning for self-driving cars by visualizing causal attention. In: Proceedings of the IEEE international conference on computer vision, Venice, Italy, 22–29 October 2017, pp. 2942–2950. New York: IEEE.
37.
AliciogluGSunB.A survey of visual analytics for explainable artificial intelligence methods. Comput Graphics2022; 102: 502–520.
38.
GunningDAhaD.Darpa’s explainable artificial intelligence (XAI) program. AI Mag2019; 40(2): 44–58.
39.
AhmedIJeonGPiccialliF.From artificial intelligence to explainable artificial intelligence in industry 4.0: a survey on what, how, and where. IEEE Trans Ind Inform2022; 18(8): 5031–5042.
40.
ZhouBKhoslaALapedrizaA, et al. Learning deep features for discriminative localization. In: Proceedings of the IEEE conference on computer vision and pattern recognition, Las Vegas, NV, USA, 27–30 June 2016, pp. 2921–2929. New York: IEEE.
41.
SelvarajuRRCogswellMDasA, et al. Grad-cam: visual explanations from deep networks via gradient-based localization. In: Proceedings of the IEEE international conference on computer vision, Venice, Italy, 22–29 October 2017, pp. 618–626. New York: IEEE.
PanYMaYDongY, et al. A vision-based monitoring method for the looseness of high-strength bolt. IEEE Trans Instrum Meas2021; 70: 1–14.
44.
HuynhTC.Vision-based autonomous bolt-looseness detection method for splice connections: design, lab-scale evaluation, and field application. Autom Constr2021; 124: 103591.
45.
YingYShichuanWWeiZ.Detection of the bolt loosening angle through semantic key point extraction detection by using an hourglass network. Struct Control Health Monit2023; 2023(1): 8860412.
46.
LuQJingYZhaoX.Bolt loosening detection using key-point detection enhanced by synthetic datasets. Appl Sci2023; 13(3): 2020.
47.
DengXLiuJGongH, et al. Detection of loosening angle for mark bolted joints with computer vision and geometric imaging. Autom Constr2022; 142: 104517.
48.
BouzidAHNechacheA.The modelling of bolted flange joints used with disc springs and tube spacers to reduce relaxation. Int J Press Vessel Pip2010; 87(12): 730–736.
RedmonJDivvalaSGirshickR, et al. You only look once: Unified, real-time object detection. In: Proceedings of the IEEE conference on computer vision and pattern recognition, Las Vegas, NV, USA, 27–30 June 2016, pp. 779–788. New York: IEEE.
51.
RedmonJFarhadiA.Yolo9000: better, faster, stronger. In: Proceedings of the IEEE conference on computer vision and pattern recognition, Honolulu, HI, USA, 21–26 July 2017, pp. 7263–7271. New York: IEEE.
52.
RedmonJFarhadiA.Yolov3: an incremental improvement. arXiv:1804.02767, 2018.
53.
ElfwingSUchibeEDoyaK.Sigmoid-weighted linear units for neural network function approximation in reinforcement learning. Neural Netw2018; 107: 3–11.
54.
BahdanauDChoKBengioY.Neural machine translation by jointly learning to align and translate. arXiv:1409.0473, 2014.
55.
VaswaniAShazeerNParmarN, et al. Attention is all you need. Adv Neural Inf Process Syst2017; 30.
56.
ZhangALiptonZCLiM, et al. Dive into deep learning. Cambridge: Cambridge University Press.
57.
WooSParkJLeeJY, et al. Cbam: convolutional block attention module. In: Proceedings of the European conference on computer vision (ECCV), 2018, pp. 3–19.
58.
ChattopadhayASarkarAHowladerP, et al. Grad-cam++: generalized gradient-based visual explanations for deep convolutional networks. In: 2018 IEEE winter conference on applications of computer vision (WACV), Lake Tahoe, NV, USA, 12–15 March 2018, pp. 839–847. New York: IEEE.
BottouL. Large-scale machine learning with stochastic gradient descent. In: Proceedings of COMPSTAT’2010: 19th international conference on computational statistics, Keynote, invited and contributed papers, Paris, France, August 22–27, 2010, pp. 177–186. Cham: Springer.
61.
MosalamKMGaoY.Artificial intelligence in vision-based structural health monitoring. Cham: Springer, 2024.
