In proportion to the immense construction of spatial structures is the emergence of catastrophes related to structural damages (e.g. loose connections), thus rendering personal injury and property loss. It is therefore essential to detect spatial bolt looseness. Current methods for detecting spatial bolt looseness mostly focus on contact-type measurement, which may not be practical in some cases. Thus, inspired by the sound-based human diagnostic approach, we develop a novel percussion method using the Mel-frequency cepstral coefficient and the memory-augmented neural network in this article. In comparison with current investigations, the main contribution of this article is the detection of multi-bolt looseness for the first time with higher accuracy than prior methods. In particular, in terms of new data obtained via similar joints, the memory-augmented neural network can help avoid inefficient relearn and assimilate new data to provide accurate prediction with only a few data samples, which effectively improves the robustness of detection. Furthermore, percussion was implemented with a robotic arm instead of manual operation, which preliminarily explores the potential of implementing automation applications in real industries. Finally, experimental results demonstrate the effectiveness of the proposed method, which can guide future development of cyber-physics systems for structural health detection.
LopezAPuenteISernaMA. Direct evaluation of the buckling loads of semi-rigidly jointed single-layer latticed domes under symmetric loading. Eng Struct2017; 29: 101–109.
5.
EbadiMDavoodiM. Evaluate axial stiffness of the mero connection, under the effect of hardening the screw. Int J Sci Emerging Tech2012; 4: 116–122.
6.
MaHFanFWenP, et al. Experimental and numerical studies on a single-layer cylindrical reticulated shell with semi-rigid joints. Thin-wall Struct2015; 86: 1–9.
7.
YangXLeiHChenYF. Constant amplitude fatigue test research on M20 high-strength bolts in grid structure with bolt-sphere joints. Adv Struct Eng2017; 20: 1466–1475.
8.
HiyamaYTakashimaHIijimaT, et al. Buckling behavior of aluminum ball Jointed single layered reticular domes. Int J Space Struct2000; 15: 81–94.
9.
ZengQGuoXHuangZ, et al. Uniaxial compression bearing capacity of bolted ball-cylinder joint. Eng Struct2019; 183: 976–986.
10.
MaHMaYYuZ, et al. Experimental and numerical research on gear-bolt joint for free-form grid spatial structures. Eng Struct2017; 148: 522–540.
11.
Moreno-GomezAAmezquita-SanchezJPValtierra-RodriguezM, et al. EMD-Shannon entropy-based methodology to detect incipient damages in a truss structure. Appl Sci2018; 8: 2068.
12.
AnYBlachowskiBZhongY, et al. Rank-revealing QR decomposition applied to damage localization in truss structures. Struct Control Health Monit2017; 24: e1849.
13.
YuLZhuJH. Structural damage prognosis on truss bridges with end connector bolts. J Eng Mech2017; 143: B4016002.
14.
GholizadASafariH. Two-dimensional continuous wavelet transform method for multidamage detection of space structures. J Perform Constr Facil2016; 30: 04016064.
15.
LiQJingX. Fault diagnosis of bolt loosening in structures with a novel second-order output spectrum–based method. Struct Health Monit2020; 19: 123–141.
16.
ZhangTBiswalSWangY. SHMnet: condition assessment of bolted connection with beyond human-level performance. Struct Health Monit. Epub ahead of print 17October2019. DOI: 10.1177/1475921719881237.
17.
HuoLWangFLiH, et al. A fractal contact theory based model for bolted connection looseness monitoring using piezoelectric transducers. Smart Mater Struct2017; 26: 104010.
18.
WangFHuoLSongG. A piezoelectric active sensing method for quantitative monitoring of bolt loosening using energy dissipation caused by tangential damping based on the fractal contact theory. Smart Mater Struct2018; 27: 015023.
19.
WangFHoSCMSongG. Monitoring of early looseness of multi-bolt connection: a new entropy-based active sensing method without saturation. Smart Mater Struct2019; 28: 10LT01.
20.
WangFHoSCMHuoL, et al. A novel fractal contact-electromechanical impedance model for quantitative monitoring of bolted joint looseness. IEEE Access2018; 6: 40212–40220.
21.
HuynhTKimJ. Quantification of temperature effect on impedance monitoring via PZT interface for prestressed tendon anchorage. Smart Mater Struct2017; 26: 125004.
22.
WangFSongG. Bolt early looseness monitoring using modified vibro-acoustic modulation by time-reversal. Mech Syst Signal Process2019; 130: 349–360.
23.
ZhangZLiuMLiaoY, et al. Contact acoustic nonlinearity (CAN)-based continuous monitoring of bolt loosening: hybrid use of high-order harmonics and spectral sidebands. Mech Syst Signal Process2018; 103: 280–294.
24.
WangFChenZSongG. Monitoring of multi-bolt connection looseness using entropy-based active sensing and genetic algorithm-based least square support vector machine. Mech Syst Signal Process2020; 136: 106507.
25.
FierroGPMMeoM. IWSHM 2017: Structural health monitoring of the loosening in a multi-bolt structure using linear and modulated nonlinear ultrasound acoustic moments approach. Struct Health Monit2018; 17: 1349–1364.
26.
YangYNgCTKotousovA. Bolted joint integrity monitoring with second harmonic generated by guided waves. Struct Health Monit2019; 18: 193–204.
27.
WangFSongG. Monitoring of multi-bolt connection looseness using a novel vibro-acoustic method. Nonlinear Dyn2020; 100: 243–254.
28.
SunHWangTLiuQ, et al. A novel eddy current array sensing film for quantitatively monitoring hole-edge crack growth in bolted joints. Smart Mater Struct2018; 28: 015018.
29.
XuJWangCLiH, et al. Health monitoring of bolted spherical joint connection based on active sensing technique using piezoceramic transducers. Sensors2018; 18: e1727.
30.
XuJDongJLiH, et al. Looseness monitoring of bolted spherical joint connection using electro-mechanical impedance technique and BP neural networks. Sensors2019; 19: 1906.
31.
RamanaLChoiWChaYJ. Fully automated vision-based loosened bolt detection using the Viola-Jones algorithm. Struct Health Monit2019; 18: 422–434.
32.
TongFXuXMLukBL, et al. Evaluation of tile-wall bonding integrity based on impact acoustics and support vector machine. Sensor Actuat A-Phys2008; 144: 97–104.
33.
Delgado-ArredondoPAMorinigo-SoteloDOsornio-RiosRA, et al. Methodology for fault detection in induction motors via sound and vibration signals. Mech Syst Signal Process2017; 83: 568–589.
34.
GlowaczA. Fault diagnosis of single-phase induction motor based on acoustic signal. Mech Syst Signal Process2019; 117: 65–80.
35.
WangFHoSCMSongG. Modeling and analysis of an impact-acoustic method for bolt looseness identification. Mech Syst Signal Process2019; 133: 106249.
36.
KongQZhuJHoSCM, et al. Tapping and listening: a new approach to bolt looseness monitoring. Smart Mater Struct2018; 27: 07LT02.
37.
ZhangYZhaoXSunX, et al. Bolt loosening detection based on audio classification. Adv Struct Eng2019; 22: 2882–2891.
38.
YuanRLvYKongQ, et al. Percussion-based bolt looseness monitoring using intrinsic multiscale entropy analysis and BP neural network. Smart Mater Struct2019; 28: 125001.
MyungHJungJJeonH. Robotic SHM and model-based positioning system for monitoring and construction automation. Adv Struct Eng2012; 15: 943–954.
41.
AdamsRDCawleyPPyeCJ, et al. A vibration technique for non-destructively assessing integrity of structures. J Mech Eng Sci1978; 20: 93–100.
42.
LeeHPhamPLargmanY, et al. Unsupervised feature learning for audio classification using convolutional deep belief networks. In: Proceedings of the 22nd international conference on neural information processing system, vol. 22, Vancouver, BC, Canada, 7–10 December 2009. NY, USA: Curran Associates Inc., pp. 1096–1104.
43.
HauDChenK. Exploring hierarchical speech representations using a deep convolutional neural network. In: Proceedings of the 11th UK workshop computational intelligence, Manchester, UK, 7–9 September 2011, http://ukci.cs.manchester.ac.uk/files/Proceedings.pdf
44.
AbdeljaberOSassiSAvciO, et al. Fault detection and severity identification of ball bearings by online condition monitoring. IEEE T Ind Electron2019; 66: 8136–8147.
45.
LiuZWuZLiT, et al. GMM and CNN hybrid method for short utterance speaker recognition. IEEE T Ind Inform2018; 14: 3244–3252.
46.
PhanHHertelLMaassM, et al. Improved audio scene classification based on label-tree embeddings and convolutional neural networks. IEEE T Audio, Speech, Lang Process2017; 25: 1278–1290.
47.
AcarEHopfgartnerFAlbayrakS. Understanding affective content of music videos through learned representations. In: International conference on multimedia modeling, Dublin, 6–10 January 2014, pp. 303–314. New York: Springer.
48.
ChowdhuryARossA. Extracting sub-glottal and supra-glottal features from MFCC using convolutional neural networks for speaker identification in degraded audio signals. In: IEEE International joint conference on biometrics (IJCB), Denver, CO, 1–4 October 2017, pp. 1–9. New York: IEEE.
49.
JinGYeBWuY, et al. Vehicle classification based on seismic signatures using convolutional neural network. IEEE Geosci Remote S2018; 16: 628–632.
50.
Abdel-HamidOMohamedARJiangH, et al. Convolutional neural networks for speech recognition. IEEE Trans Audio, Speech, Lang Process2014; 22: 1533–1545.
51.
SehgalAKehtarnavazN. A convolutional neural network smartphone app for real-time voice activity detection. IEEE Access2018; 6: 9017–9026.
52.
NodaKYamaguchiYNakadaiK, et al. Audio-visual speech recognition using deep learning. Appl Intell2015; 42: 722–737.
53.
LiTLHChanABChunAHW. Automatic musical pattern feature extraction using convolutional neural network. In: Proceedings of the international multiconference of engineers and computer scientists 2010, IMECS, Kowloon, Hong Kong, 17–19 March 2010, pp. 546–550. Kowloon, Hong Kong: City University of Hong Kong.
54.
Eghbal-zadehHLehnerBDorferM, et al. A hybrid approach with multi-channel I-vectors and convolutional neural networks for acoustic scene classification, 2017, https://arxiv.org/abs/1706.06525
55.
YuYTangSRaposoF, et al. Deep cross-modal correlation learning for audio and lyrics in music retrieval. ACM T Multim Comput2019; 15: 20.
56.
JiangFLiHZhangZ, et al. An event recognition method for fiber distributed acoustic sensing systems based on the combination of MFCC and CNN. In: International conference on optical instruments and technology: advanced optical sensors and applications, vol. 10618, Beijing, China, 28–30 October 2017. Bellingham, WA: SPIE.
57.
HuangHMChenWKLiuCH, et al. Singing voice detection based on convolutional neural networks. In: 7th international symposium on next generation electronics (ISNE), Taipei, 7–9 May 2018. New York: IEEE.
58.
BahouraM. Pattern recognition methods applied to respiratory sounds classification into normal and wheeze classes. Comput Biol Med2009; 39: 824–843.
59.
SandipanCAnindyaRGoutamS. Improved closed set text-independent speaker identification by combining MFCC with evidence from flipped filter banks. Int J Signal Process2007; 4: 114–122.
60.
AiOCHariharanMYaacobS, et al. Classification of speech dysfluencies with MFCC and LPCC features. Expert Syst Appl2012; 39: 2157–2165.
HochreiterSYoungerASConwellPR. Learning to learn using gradient descent. In: Proceedings of the international conference on artificial neural networks (ICANN), Vienna, 21–25 August 2001, pp. 87–94. New York: Springer.
GopalanRLiRChellappaR. Domain adaptation for object recognition: an unsupervised approach. In: Proceedings of the international conference on computer vision, Barcelona, 6–13 November 2011, pp. 999–1006. New York: IEEE.
67.
MobinyAMoulikSNguyenHV. Lung cancer screening using adaptive memory-augmented recurrent networks, 2017, https://arxiv.org/abs/1710.05719
68.
GravesAWayneGReynoldsM, et al. Hybrid computing using a neural network with dynamic external memory. Nature2016; 538: 471–476.
69.
YuSIndurthiSBackS, et al. A multi-stage memory augmented neural network for machine reading comprehension. In: Proceedings of the workshop on machine reading for question answering, 2018, pp. 21–30. Association for Computational Linguistics, https://www.aclweb.org/anthology/W18-2603.pdf
70.
BerceaCIPaulyOMaierA, et al. SHAMANN: shared memory augmented neural networks. In: International conference on information processing in medical imaging, Hong Kong, China, 2–7 June 2019, pp. 830–841.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
