Machine vision-based structural health monitoring is gaining popularity due to the rich information one can extract from video and images. However, the extraction of characteristic parameters from images often requires manual intervention, thereby limiting its scalability and effectiveness. In contrast, deep learning overcomes the aforementioned shortcoming in that it can autonomously extract feature parameters (e.g. structural damage) from image datasets. Therefore, this study aims to validate the use of machine vision and deep learning for structural health monitoring by focusing on a particular application of detecting bolt loosening. First, a dataset that contains 300 images was collected. The dataset includes two bolt states, namely, tight and loosened. Second, a faster region-based convolutional neural network was trained and evaluated. The test results showed that the average precision of bolt damage detection is 0.9503. Thereafter, bolts were loosened to various screw heights, and images obtained from different angles, lighting conditions, and vibration conditions were identified separately. The trained model was then employed to validate that bolt loosening could be detected with sufficient accuracy using various types of images. Finally, the trained model was connected with a webcam to realize real-time bolt loosening damage monitoring.
DoeblingSWFarrarCPrimeMB, et al. Damage identification and health monitoring of structural and mechanical systems from changes in their vibration characteristics: a literature review. Shock Vib Dig1996; 30(11): 2043–2049.
2.
HousnerGWBergmanLACaugheyTK, et al. Structural control: past, present, and future. J Eng Mech1997; 123(9): 897–971.
ZhouG-DYiT-H.Recent developments on wireless sensor networks technology for bridge health monitoring. Math Probl Eng2013; 2013(3): 947867.
5.
FedericiFGraziosiFFaccioM, et al. An integrated approach to the design of wireless sensor networks for structural health monitoring. Int J Distrib Sens N. Epub ahead of print 22 March 2012. DOI: 10.1155/2012/594842.
6.
OhJKJangGOhS, et al. Bridge inspection robot system with machine vision. Automat Constr2009; 18(7): 929–941.
7.
KimSWJeonBGKimNS, et al. Vision-based monitoring system for evaluating cable tensile forces on a cable-stayed bridge. Struct Health Monit2013; 12(5–6): 440–456.
8.
LuoLFengMQWuZY.Robust vision sensor for multi-point displacement monitoring of bridges in the field. Eng Struct2018; 163: 255–266.
9.
TrivediMMGandhiTMcCallJ.Looking-in and looking-out of a vehicle: computer-vision-based enhanced vehicle safety. IEEE T Intell Transp2007; 8(1): 108–120.
10.
ZhuZXuGYangB, et al. VISATRAM: a real-time vision system for automatic traffic monitoring. Image Vision Comput2000; 18(10): 781–794.
11.
Zaczek-PeplinskaJElżbieta KowalskaMMalowanyK, et al. Application of digital image correlation and geodetic displacement measuring methods to monitor water dam behavior under dynamic load. J Civ Eng Archit2015; 9(12): 1496–1505.
12.
JahanshahiMRMasriSF. A novel vision-based crack quantification approach by incorporating depth perception for condition assessment of structures. In: International conference on computing in civil engineering, Clearwater Beach, FL, 17–20 June 2012, pp. 531–536. Reston, VA: ASCE.
JahanshahiMRMasriSF.Adaptive vision-based crack detection using 3D scene reconstruction for condition assessment of structures. Automat Constr2012; 22: 567–576.
15.
LiRZhangWSukHI, et al. Deep learning based imaging data completion for improved brain disease diagnosis. Med Image Comput Comput Assist Interv2014; 17(Pt 3): 305–312.
16.
NieLWangMZhangL, et al. Disease inference from health-related questions via sparse deep learning. IEEE T Knowl Data En2015; 27(8): 2107–2119.
17.
LiuSLiuSCaiW, et al. Early diagnosis of Alzheimer’s disease with deep learning. In: IEEE international symposium on biomedical imaging, Beijing, China, 29 April–2 May 2014, pp. 1015–1018. New York: IEEE.
18.
HammerlaNYFisherJMAndrasP, et al. PD disease state assessment in naturalistic environments using deep learning. In: Twenty-ninth AAAI conference on artificial intelligence, Austin, TX, 25–30 January 2015, pp. 1742–1748. Palo Alto, CA: AAAI Press.
19.
WlodarczakPSoarJAllyM.Multimedia data mining using deep learning. In: Fifth international conference on digital information processing and communications, Sierre, 7–9 October 2015, pp. 190–196. New York: IEEE.
20.
LiuMNiuJWangX. An autopilot system based on ROS distributed architecture and deep learning. In: IEEE international conference on industrial informatics, Emden, 24–26 July 2017, pp. 1229–1234. New York: IEEE.
FaisalMManzoorS.Deep learning for lip reading using audio-visual information for Urdu language, 2018, https://arxiv.org/abs/1802.05521
26.
ChaYJChoiW.Vision-based concrete crack detection using a convolutional neural network. In: CaicedoJPakzadS (eds) Dynamics of civil structures, vol. 2. Cham: Springer, 2017, pp. 71–73.
27.
ParkJHHuynhTCChoiSH, et al. Vision-based technique for bolt-loosening detection in wind turbine tower. Wind Struct2015; 21(6): 709–726.
28.
RamanaLChoiWChaYJ.Fully automated vision-based loosened bolt detection using the Viola–Jones algorithm. Struct Health Monit2018; 18: 422–434.
29.
RamanaLChoiWChaYJ. Automated vision-based loosened bolt detection using the cascade detector. In: Wee SitEWalberCWalterP, et al. (eds) Sensors and instrumentation, vol. 5. Cham: Springer, 2017, pp. 23–28.
30.
GirshickRDonahueJDarrellT, et al. Rich feature hierarchies for accurate object detection and semantic segmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition, Columbus, OH, 23–28 June 2014.
31.
RenSHeKGirshickR, et al. Faster R-CNN: towards real-time object detection with region proposal networks. In: CortesCLawrenceNDLeeDD, et al. (eds) Advances in neural information processing systems, vol. 28. Red Hook, NY: Curran Associates, Inc., 2015, pp. 91–95.
