Machine reliability is of great importance in modern industries, where timely, accurate, and explainable fault diagnosis based on vibration data is one of the key processes. Traditional contact vibration sensors face installation limits and signal distortion. The existing intelligent fault diagnosis methods are typically not explainable. In recent years, event-based cameras offer a promising noncontact solution due to their asynchronous recording and high speed. Meanwhile, large models are able to provide expert explanations in many fields. To address the aforementioned problems in fault diagnosis, we propose a physics-informed multimodal large model with dynamic vision in this paper. First, we build videos from dynamic vision data and use frequency-domain physical priors to guide fault reasoning. Meanwhile, a separate classification head is proposed to improve both fault diagnosis and explainable text generation effects. Experiments show the proposed method is not only as accurate as the contact vibration sensing methods but also provides reasonable expert explanations for machine fault diagnosis. With the high interpretability, flexibility, and accuracy, the proposed method is thus validated as a promising new tool for machine fault diagnosis.
CherukuSBalajiSDelgadoA, et al. Data-driven digital twins for real-time machine monitoring: a case study on a rotating machine. J Comput Inform Sci Eng2025; 25(3): 031005.
2.
ZhangWXuMYangH, et al. Data-driven deep learning approach for thrust prediction of solid rocket motors. Measurement2024; 225: 114051.
3.
SpirtoMNicolellaAMellusoF, et al. Enhancing SDP-CNN for gear fault detection under variable working conditions via multi-order tracking filtering. J Dyn Monit Diagn2025; 4(4): 226–238.
4.
KharkovskySZoughiR. Microwave and millimeter wave nondestructive testing and evaluation – overview and recent advances. IEEE Instrum Meas Mag2007; 10(2): 26–38.
5.
ZabihiAAghdasiFEllouziC, et al. Non-contact wind turbine blade crack detection using laser doppler vibrometers. Energies2024; 17(9): 2165.
6.
International Organization for Standardization. ISO 5348:2021: mechanical vibration and shock—mechanical mounting of accelerometers. Standard, 2021.
7.
ChenSBekarETBokrantzJ, et al. Ai-enhanced digital twins in maintenance: systematic review, industrial challenges, and bridging research–practice gaps. J Manuf Syst2025; 82: 678–699.
8.
AlotaibiMAsliBHSKhanM. Non-invasive inspections: a review on methods and tools. Sensors2021; 21(24): 8474.
9.
ZhaoWXZhouKLiJ, et al. A survey of large language models. arXiv:230318223, 2023.
10.
ZhouXDuanPXiaokaitiY, et al. Event-based visual vibrometry. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). 2025. pp. 24666–24676. Piscataway, NJ: IEEE.
11.
YuSLiXLeiY, et al. Dynamic vision-enabled intelligent micro-vibration estimation method with spatiotemporal pattern consistency. IEEE/CAA J Autom Sin2025; 12(11): 2359–2361.
12.
LiXYuSChenX, et al. XJTU-DV: open-source dynamic vision dataset for non-contact vibration measurement and fault diagnosis of mechanical systems. Chin J Mech Eng2025; 39: 100163.
13.
GuangRLiXLeiY, et al. Dynamic vision-based machine vibration sensing and fault diagnosis with signal alignment and feature clustering. Eng Appl Artif Intell2025; 162: 112445.
LiuCLiXChenX, et al. Neuromorphic computing-enabled generalized machine fault diagnosis with dynamic vision. Adv Eng Inform2025; 65: 103300.
16.
LeiYYangBJiangX, et al. Applications of machine learning to machine fault diagnosis: a review and roadmap. Mech Syst Signal Process2020; 138: 106587.
17.
JiaNHuangWDingC, et al. Physics-informed unsupervised domain adaptation framework for cross-machine bearing fault diagnosis. Adv Eng Inform2024; 62: 102774.
18.
ZhangWLiXMaH, et al. Transfer learning using deep representation regularization in remaining useful life prediction across operating conditions. Reliab Eng Syst Saf2021; 211: 107556.
19.
WangRShaoYLiQ, et al. A novel transformer-based semantic segmentation framework for structural condition assessment. Struct Health Monit2024; 23(2): 1170–1183.
20.
WangTLiYTianH, et al. Selective knowledge distillation-based domain adaptation framework towards edge computing fault diagnosis for high-speed train bogie. Green Energy Intell Transp2025; 5(4): 100373.
21.
ZhangWLiXMaH, et al. Open-set domain adaptation in machinery fault diagnostics using instance-level weighted adversarial learning. IEEE Trans Ind Inform2021; 17(11): 7445–7455.
22.
XueYWenCWangZ, et al. A novel framework for motor bearing fault diagnosis based on multi-transformation domain and multi-source data. Knowl Based Syst2024; 283: 111205.
23.
WanLNingJLiY, et al. Intelligent fault diagnosis via ring-based decentralized federated transfer learning. Knowl Based Syst2024; 284: 111288.
24.
ZhangWWangZLiX. Blockchain-based decentralized federated transfer learning methodology for collaborative machinery fault diagnosis. Reliab Eng Syst Saf2023; 229: 108885.
25.
ZhangWLiX. Data privacy preserving federated transfer learning in machinery fault diagnostics using prior distributions. Struct Health Monit2022; 21(4): 1329–1344.
26.
ZhangWJiangNYangS, et al. Federated transfer learning for remaining useful life prediction in prognostics with data privacy. Meas Sci Technol2025; 36(7): 076107.
27.
WuYSicardBGadsdenSA. Physics-informed machine learning: a comprehensive review on applications in anomaly detection and condition monitoring. Expert Syst Appl2024; 255: 124678.
28.
ZargarSAYuanFG. Physics-informed deep learning for scattered full wavefield reconstruction from a sparse set of sensor data for impact diagnosis in structural health monitoring. Struct Health Monit2024; 23(5): 2963–2979.
29.
SongYShanSZhangY, et al. Physics-guided neural network for structural health monitoring with lamb waves through boundary reflection elimination. Struct Health Monit2026; 25(2): 1219–1236.
30.
YangQTangBDengL, et al. A hybrid physics-corrected neural network for RUL prognosis under random missing data. Expert Syst Appl2024; 256: 124939.
31.
PengCShengYGuiW, et al. Bearing fault diagnosis based on multimodal knowledge graphs under few-shot samples. Knowl Based Syst2025; 330: 114604.
32.
GargPKeprateASoniG, et al. Fault detection in wind turbine bearings by coupling knowledge graph and machine learning approach. J Dyn Monit Diagn2025; 4(4): 250–263.
33.
QinYLiuHWangY, et al. Inverse physics–informed neural networks for digital twin–based bearing fault diagnosis under imbalanced samples. Knowl Based Syst2024; 292: 111641.
34.
ZhangWLiXLiX. Deep learning-based prognostic approach for lithium-ion batteries with adaptive time-series prediction and on-line validation. Measurement2020; 164: 108052.
35.
ZhangWHaoHZhangY, et al. State of charge prediction for lithium-ion batteries in electric aircraft based on self-supervised informer. Appl Soft Comput2025; 186: 114283.
36.
LiWHuangRLiJ, et al. A perspective survey on deep transfer learning for fault diagnosis in industrial scenarios: theories, applications and challenges. Mech Syst Signal Process2022; 167: 108487.
37.
PanLLiGZhuT, et al. Physics-informed machine learning in design and manufacturing: status and challenges. J Comput Inform Sci Eng2025; 25: 1–29.
38.
XuZZhaoKWangJ, et al. Physics-informed probabilistic deep network with interpretable mechanism for trustworthy mechanical fault diagnosis. Adv Eng Inform2024; 62: 102806.
ZhaoSLiuSJiangY, et al. Industrial foundation models (IFMS) for intelligent manufacturing: a systematic review. J Manuf Syst2025; 82: 420–448.
41.
YuSLiXLeiY, et al. Multimodal data-enabled large model for machine fault diagnosis towards intelligent operation and maintenance. J Ind Inform Integr2026; 50: 101061.
42.
ZhangQXuCLiJ, et al. Llm-tsfd: an industrial time series human-in-the-loop fault diagnosis method based on a large language model. Expert Syst Appl2025; 264: 125861.
43.
LuZSunCLiX. Multimodal large language model-enabled machine intelligent fault diagnosis method with non-contact dynamic vision data. Sensors2025; 25(18): 5898.
44.
ZhaoMShenXJiangF. Research on mechanical vibration measurement method based on event camera. In: 2023 3rd International conference on energy engineering and power systems (EEPS), Dali, China, 28–30 July, 2023, pp. 528–532. Piscataway, NJ: IEEE.
45.
ShiFChenLZhouM, et al. A stepwise intelligence generative method for structured maintenance guidance documents based on knowledge graph augmented LLM. Adv Eng Inform2025; 67: 103523.
46.
SheaKStankovićTAgrawalA, et al. Expanding the generative power of large language models (LLMs) for design through formal design grammars and languages. J Comput Inform Sci Eng2025; 25(12): 120802.
47.
HuangLYuWMaW, et al. A survey on hallucination in large language models: principles, taxonomy, challenges, and open questions. ACM Trans Inf Syst2025; 43(2): 1–55.
48.
LiYZhaoHJiangH, et al. Large language models for manufacturing. J Manuf Syst2026; 86: 516–545.
49.
YinWHayJRothD. Benchmarking zero-shot text classification: datasets, evaluation and entailment approach. arXiv:190900161, 2019.
50.
GuangRLiXLeiY, et al. Non-contact machine vibration sensing and fault diagnosis method based on event camera. In: 2023 CAA symposium on fault detection, supervision and safety for technical processes (SAFEPROCESS), Yibin, China, September 2023, pp. 1–5. Piscataway, NJ, USA: IEEE.
51.
NaWjSunKHJeonBC, et al. Event-based micro vibration measurement using phase correlation template matching with event filter optimization. Measurement2023; 215: 112867.
52.
ChenLLiJDongX, et al. Sharegpt4v: improving large multi-modal models with better captions. In: European conference on computer vision (ECCV). Cham, Switzerland: Springer, 2024, pp. 370–387.
53.
LiuHLiCWuQ, et al. Visual instruction tuning. Adv Neural Inform Process Syst2023; 36: 34892–34916.
54.
DevlinJChangMWLeeK, et al. Bert: pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, vol. 1 (long and short papers), Minneapolis, MN, USA, June 2019, pp. 4171–4186. Stroudsburg, PA, USA: ACL.
55.
WolfTDebutLSanhV, et al. Transformers: state-of-the-art natural language processing. In: Proceedings of the 2020 conference on empirical methods in natural language processing: system demonstrations, 2020, pp. 38–45.
56.
BaiSChenKLiuX, et al. Qwen2. 5-vl technical report. arXiv:250213923, 2025.
57.
XuJGuoZHeJ, et al. Qwen2. 5-omni technical report. arXiv:250320215, 2025.
58.
HuEJShenYWallisP, et al. Lora: low-rank adaptation of large language models. ICLR2022; 1(2): 3.
59.
WeiJWangXSchuurmansD, et al. Chain-of-thought prompting elicits reasoning in large language models. Adv Neural Inform Process Syst2022; 35: 24824–24837.
60.
WuJGanWChenZ, et al. Ai-generated content (AIGC): a survey. arXiv:230406632, 2023.
61.
LiXZhangWMaH, et al. Degradation alignment in remaining useful life prediction using deep cycle-consistent learning. IEEE Trans Neural Netw Learn Syst2021; 33(10): 5480–5491.
