Driver fatigue has long been recognized as a major cause of severe traffic accidents. However, achieving precise real-time fatigue detection for in-cabin drivers using low-cost devices remains a significant challenge. To tackle this issue, we propose a fatigue detection system for drivers based on temporal facial features. In the initial phase, we integrate a high-precision landmark model trained using a teacher-student distillation model with the You Only Look Once 5 face (YOLO5face) model to rapidly and accurately extract facial landmarks. Subsequently, a series of specific criteria are defined for classifying sequential parameters related to eye movement, mouth activity, and head posture. Extensive validation on both self-built dataset and four publicly available datasets demonstrates that both eye and head poses can be accurately detected, achieving an impressive accuracy rate of 97.1% in yawning detection. Furthermore, the system meets real-time detection requirements, operating at inference speed of 8–20 ms on a standard CPU. The source code is available at: https://github.com/Benpowder/Temporal-Facial-Features-Based-Fatigue-Detection-System.
ZhangGYauKKWZhangX, et al. Traffic accidents involving fatigue driving and their extent of casualties. Accid Anal Prev2016; 87: 34–42.
2.
QuYHuHLiuJ, et al. Driver state monitoring technology for conditionally automated vehicles: review and future prospects. IEEE Trans Instrum Meas2023; 72(3000920): 1–20.
3.
LawoyinSFeiDYBaiO.Accelerometer-based steering-wheel movement monitoring for drowsy-driving detection. Proc IMechE, Part D: J Automobile Engineering2015; 229(2): 163–173.
4.
JiaoYChenXSunZ, et al. Data-driven detection and assessment for urban railway transit driver fatigue in real work conditions. Proc IMechE, Part D: J Automobile Engineering2023; 2667(1): 360–369.
5.
JungSJooSOhC.Evaluating the effects of supplemental rest areas on freeway crashes caused by drowsy driving. Accid Anal Prev2017; 99: 356–363.
6.
IbrahimMMSoraghanJJPetropoulakisL, et al. Yawn analysis with mouth occlusion detection. Biomed Signal Process Control2015; 18: 360–369.
7.
FeiYLiBWangB, et al. Long short-term memory network based fatigue detection with sequential mouth feature. In: 2020 international symposium on autonomous systems, December 2020, pp.17–22. DOI: 10.1109/ISAS49493.2020.9378877.
8.
ShabnamAMonaOShervinS, et al. YawDD: yawning detection dataset. In: Proceedings of the 5th ACM multimedia systems conference, March 2014, pp.24–28. DOI: 10.21227/e1qm-hb90.
9.
BaiJYuWXiaoZ, et al. Two-stream spatial-temporal graph convolutional networks for driver drowsiness detection. IEEE Trans Cybern2022; 52(12): 13821–13833.
10.
WangJWangYDaiY, et al. Cooperative detection method for distracted and fatigued driving behaviors with readily embedded system implementation. IEEE Trans Instrum Meas2022; 71(3000713): 1–13.
11.
D’OrazioTLeoMGuaragnellaC, et al. A visual approach for driver inattention detection. Pattern Recognit2007; 40(8): 2341–2355.
12.
GarcíaSBronteLMBergasaJ, et al. Vision-based drowsiness detector for real driving conditions. In: IEEE intelligent vehicles symposium, June 2012, pp.618–623. DOI: 10.1109/IVS.2012.6232222.
13.
LiXXiaJCaoL, et al. Driver fatigue detection based on convolutional neural network and face alignment for edge computing device. Proc IMechE, Part D: J Automobile Engineering2021; 235(10–11): 2699–2711.
14.
SinghNTPathakNRaizadaA, et al. Real-time driver drowsiness detection system using cascaded ConvNet framework. In: 2023 international conference on sustainable computing and smart systems, June 2023, pp.828–833. DOI: 10.1109/ICSCSS57650.2023.10169434.
15.
ZhangZLuoPLoyCC, et al. Learning deep representation for face alignment with auxiliary attributes. IEEE Trans Pattern Anal Mach Intell2016; 38(5): 918–930.
16.
MandalBLiLWangGS, et al. Towards detection of bus driver fatigue based on robust visual nalysis of eye state. IEEE Trans Intell Transp Syst2017; 18(3): 545–557.
17.
DongXYuSIWengX, et al. Supervision-by-registration: an unsupervised approach to improve the precision of facial landmark detectors. In: IEEE conference on computer vision and pattern recognition, June 2018, pp.360–368. DOI: 10.1109/CVPR.2018.00045.
18.
WuYJiQ.Facial landmark detection: a literature survey. Int J Comput Vis2019; 127(2): 115–142.
19.
AbbasiJAMullinsDRingelsteinN, et al. An analysis of driver gaze behaviour at roundabouts. IEEE Trans Intell Transp Syst2022; 23(7): 8715–8724.
20.
MaiorCBSMouraMJDCSantanaJMM, et al. Real-time classification for autonomous drowsiness detection using eye aspect ratio. Expert Syst Appl2020; 158(158): 113505.
21.
QuddusAShahidi ZandiAPrestL, et al. Using long short term memory and convolutional neural networks for driver drowsiness detection. Accid Anal Prev2021; 156: 106107.
22.
LiWHuangJXieG, et al. A survey on vision-based driver distraction analysis. J Syst Archit2021; 121: 102319.
23.
HariCVSankaranP.Driver distraction analysis using face pose cues. Expert Syst Appl2021; 179: 115036.
24.
MartinM, et al. Drive&act: a multi-modal dataset for fine-grained driver behavior recognition in autonomous vehicles. In: Proceedings of the IEEE international conference on computer vision, October 2019, pp.2801–2810. DOI: 10.1109/ICCV.2019.00289.
25.
YangTYChenYTLinYY, et al. FSA-Net: learning fine-grained structure aggregation for head pose estimation from a single image. In: IEEE conference on computer vision and pattern recognition, June 2019, pp.1087–1096.
26.
AnsariSNaghdyFDuH, et al. Driver mental fatigue detection based on head posture using new modified RELU-BiLSTM deep neural network. IEEE Trans Intell Transp Syst2022; 23(8): 10957–10969.
27.
LuYLiuCChangF, et al. JHPFA-Net: joint head pose and facial action network for driver yawning detection across arbitrary poses in videos. IEEE Trans Intell Transp Syst2023; 24(11): 11850–11863.
28.
YangGLinYBhattacharyaP.A driver fatigue recognition model based on information fusion and dynamic Bayesian network. Inf Sci2010; 180(10): 1942–1954.
29.
SunYYuXB.An innovative nonintrusive driver assistance system for vital signal monitoring. IEEE J Biomed Health Inform2014; 18(6): 1932–1939.
30.
BakkerBZablockiBBakerA, et al. A multi-stage, multi-feature machine learning approach to detect driver sleepiness in naturalistic road driving conditions. IEEE Trans Intell Transp Syst2022; 23(5): 4791–4800.
31.
DuGLiTLiC, et al. Vision-based fatigue driving recognition method integrating heart rate and facial features. IEEE Trans Intell Transp Syst2021; 22(5): 3089–3100.
32.
SunZMiaoYJeonJY, et al. Facial feature fusion convolutional neural network for driver fatigue detection. Eng Appl Artif Intell2023; 126: 106981.
33.
ShahbakhtiMBeiramvandMNasiriE, et al. Fusion of EEG and eye blink analysis for detection of driver fatigue. IEEE Trans Neural Syst Rehabil Eng2023; 31: 2037–2046.
34.
LiuFChenDZhouJ, et al. A review of driver fatigue detection and its advances on the use of RGB-D camera and deep learning. Eng Appl Artif Intell2022; 116: 105399.
35.
YangSLuoPLoyCC, et al. Wider face: a face detection benchmark. In: Proceedings of the IEEE conference on computer vision and pattern recognition, November 2016, pp.5525–5533. DOI: 10.1109/CVPR.2016.596.
36.
LiuDYamasakiTWangY, et al. Toward extremely lightweight distracted driver recognition with distillation-based neural architecture search and knowledge transfer. IEEE Trans Intell Transp Syst2023; 24(1): 764–777.
37.
WengCLaiYLaiS. Driver drowsiness detection via a hierarchical temporal deep belief network. In: Computer vision–ACCV 2016 workshops: ACCV 2016 international workshops, November 2016, vol. 3, no. 13, pp.117–133. DOI: 10.1007/978-3-319-54526-49.
38.
MassozQLangohrTFrançoisC, et al. The ULg multimodality drowsiness database (called DROZY) and examples of use. In: 2016 IEEE winter conference on applications of computer vision (WACV), March2016, pp.1–7. DOI: 10.1109/WACV.2016.7477715.
39.
GhoddoosianRGalibMAthitsosV. A realistic dataset and baseline temporal model for early drowsiness detection. In: IEEE/CVF conference on computer vision and pattern recognition workshops (CVPRW), April 2019, pp.178–187. DOI: 10.1109/CVPRW.2019.00027.
