The imaging photoplethysmography (iPPG) method is a non-invasive, non-contact measurement method that uses a camera to detect physiological indicators. On the other hand, wearing a mask has become essential today when COVID-19 is rampant, which has become a new challenge for heart rate (HR) estimation from facial videos recorded by a camera.
OBJECTIVE:
The aim is to propose an iPPG-based method that can accurately estimate HR with or without a mask.
METHODS:
First, the facial regions of interest (ROI) were divided into two sub-ROIs, and the original signal was obtained through spatial averaging with different weights according to the result of judging whether wearing a mask or not, and the CDF, which emphasizes the main component signal, was combined with the improved POS suitable for real-time HR estimation to obtain the noise-removed BVP signal.
RESULTS:
For self-collected data while wearing a mask, MAE, RMSE, and ACC were 1.09 bpm, 1.44 bpm, and 99.08%, respectively.
CONCLUSION:
Experimental results show that the proposed framework can estimate HR stably in real-time in both cases of wearing a mask or not. This study expands the application range of HR estimation based on facial videos and has very practical value in real-time HR estimation in daily life.
SeravalleGTrevanoFQGrassiG. Heart rate as a predictor of cardiovascular risk. Minerva Med.2020; 112(1): 130-143. doi: 10.23736/S0026-4806.20.06695-1.
2.
MayRWSeibertGSSanchez-GonzalezMAFinchamFD. School burnout and heart rate variability: Risk of cardiovascular disease and hypertension in young adult females. Stress.2018; 21(3): 211-216. doi: 10.1080/10253890.2018.1433161.
3.
ChengCHWongKLChinJWChanTTSoRHY. Deep learning methods for remote heart rate measurement: A review and future research agenda. Sensors.2021; 21: 6296. doi: 10.3390/s21186296.
FaizalAThevarajahTMKhorSMChangSW. A review of risk prediction models in cardiovascular disease: conventional approach vs. artificial intelligent approach. Comput. Meth. Programs Biomed.2021; 207(25): 106190. doi: 10.1016/j.cmpb.2021.106190.
6.
KhanamFAl-NajiAChahlJ. Remote monitoring of vital signs in diverse non-clinical and clinical scenarios using computer vision systems: A review. Appl. Sci.2019; 9(20): Art. no. 4474. doi: 10.3390/app9204474.
7.
WuTBlazekVSchmittHJ. Photoplethysmography imaging: A new noninvasive and noncontact method for mapping of the dermal perfusion changes. Proc. SPIE-Int. Soc. Opt. Eng.2000; 4163: 62-70. doi: 10.1117/12.407646.
RyuJHongSLiangSPakSChenQYanS. A new framework for robust heart rate measurement based on the head motion state estimation. IEEE J. Biomed. Health Inform.2021; 25(9): 3428-3437. doi: 10.1109/JBHI.2021.3083917.
10.
RyuJHongSLiangSPakSChenQYanS. A measurement of illumination variation-resistant noncontact heart rate based on the combination of singular spectrum analysis and sub-band method. Comput. Meth. Programs Biomed.2021; 200(3): Art. No. 105824. doi: 10.1016/j.cmpb.2020.105824.
11.
PohMZMcDuffDJPicardRW. Non-contact, automated cardiac pulse measurements using video imaging and blind source separation. Opt. Express.2010; 18(10): 10762-10774. doi: 10.1364/OE.18.010762.
12.
ZhaoFLiMQianYTsienJZ. Remote measurements of heart and respiration rates for telemedicine. PLoS One.2013; 8(10): Art. No. e71384. doi: 10.1371/journal.pone.0071384.
13.
PohMZMcDuffDJPicardRW. Advancements in noncontact, multiparameter physiological measurements using a webcam. IEEE Trans. Biomed. Eng.2011; 58(1): 7-11. doi: 10.1109/TBME.2010.2086456.
14.
LewandowskaMRuminskiJKocejkoTNowakJ. Measuring pulse rate with a webcam – a non-contact method for evaluating cardiac activity. In Federated Conference on Computer Science and Information Systems (FedCSIS), Szczecin, Poland. 2011; pp. 18-21.
15.
HassanHJaidkaSDwyerVMHuS. Assessing blood vessel perfusion and vital signs through retinal imaging photoplethysmography. Biomed. Opt. Express.2018; 9(5): 2351-2364. doi: 10.1364/BOE.9.002351.
16.
GuoZWangZJShenZ. Physiological parameter monitoring of drivers based on video data and independent vector analysis. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Florence, Italy. 2014; pp. 4374-4378. doi: 10.1109/ICASSP.2014.6854428.
17.
QiHGuoZChenXShenZWangZJ. Video-based human heart rate measurement using joint blind source separation. Biomed. Signal Process. Control.2017; 31: 309-320. doi: 10.1016/j.bspc.2016.08.020.
18.
XieKFuCLiangHHongHZhuX. Non-contact Heart Rate Monitoring for Intensive Exercise Based on Singular Spectrum Analysis. In IEEE Conference on Multimedia Information Processing and Retrieval (MIPR), San Jose, CA, USA. 2019; pp. 228-233. doi: 10.1109/MIPR.2019.00048.
19.
WangDYangXLiuXJingJFangS. Detail-preserving pulse wave extraction from facial videos using consumer-level camera. Biomed. Opt. Express.2020; 11(4): 1876-1891. doi: 10.1364/BOE.380646.
20.
LinKChenDTsaiW. Face-based heart rate signal decomposition and evaluation using multiple linear regression. IEEE Sens. J.2016; 16(5): 1351-1360. doi: 10.1109/JSEN.2015.2500032.
WangZKYingKHsuCT. Vision-Based Heart Rate Estimation Via A Two-Stream CNN. In IEEE International Conference on Image Processing (ICIP), Taipei, Taiwan. 2019; pp. 3327-3331. doi: 10.1109/ICIP.2019.8803649.
25.
NiuXShanSHanHChenX. RhythmNet: End-to-end heart rate estimation from face via spatial-temporal representation. IEEE Trans. Image Process.2020; 29: 2409-2423. doi: 10.1109/TIP.2019.2947204.
26.
WangWStuijkSde HaanG. A novel algorithm for remote photoplethysmography: Spatial subspace rotation. IEEE Trans. Biomed. Eng.2016; 63(9): 1974-1984. doi: 10.1109/TBME.2015.2508602.
27.
de HaanGJeanneV. Robust pulse rate from chrominance-based rPPG. IEEE Trans. Biomed. Eng.2013; 60(10): 2878-2886. doi: 10.1109/TBME.2013.2266196.
28.
de HaanGvan LeestA. Improved motion robustness of remote-PPG by using the blood volume pulse signature. Physiol. Meas.2014; 35(9): 1913-1926. doi: 10.1088/0967-3334/35/9/1913.
RyuJHongSLiangSPakSChenQYanS. Research on the combination of color channels in heart rate measurement based on photoplethysmography imaging. J. Biomed. Opt.2021; 26(2): Art. no. 025003. doi: 10.1117/1.JBO.26.2.025003.
31.
RumińskiJ. Reliability of pulse measurements in videoplethysmography. Metrol. Meas. Syst.2016; 23(3): 359-371. doi: 10.1515/mms-2016-0040.
ViolaPJonesM. Rapid object detection using a boosted cascade of simple features. In Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR), Kauai, HI, USA. 2001; pp. 1511-1518. doi: 10.1109/CVPR.2001.990517.
34.
KingDE. Dlib-ml: A machine learning toolkit. J. Mach. Learn. Res.2009; 10(3): 1755-1758. doi: 10.1145/1577069.1755843.
35.
KartynnikYAblavatskiAGrishchenkoIGrundmannM. Real-time Facial Surface Geometry from Monocular Video on Mobile GPUs. 2019: arXiv:1907.06724.
36.
AIZOO. Face-Mask detection [Source code]. 2021. Available from: https//github.com/AIZOOTech/FaceMaskDetection.
37.
StrickerRMüllerSGrossH. Non-contact video-based pulse rate measurement on a mobile service robot. In the 23rd IEEE International Symposium on Robot and Human Interactive Communication, Edinburgh, UK. 2014; pp. 1056-1062. doi: 10.1109/ROMAN.2014.6926392.
WangWden BrinkerACStuijkSde HaanG. Color-Distortion Filtering for Remote Photoplethysmography. In 2017 12th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2017), Washington, DC, USA. 2017; pp. 71-78. doi: 10.1109/FG.2017.18.
43.
BadnjevićAPokviécLGDžemićZBečićF. Risks of emergency use authorizations for medical products during outbreak situations: A COVID-19 case study. Biomed. Eng. Online.2020; 19(1): Art. no. 75. doi: 10.1186/s12938-020-00820-0.
44.
BadnjevićAAvdihodžićHGurbeta PokvićL. Artificial intelligence in medical devices: Past, present and future. Psychiatr Danub.2021; 33(3): S336-S341.
