Human respiration rate (HRR) is an important physiological metric for diagnosing a variety of health conditions from stress levels to heart conditions. Estimation of HRR is well-studied in controlled terrestrial environments, yet robotic estimation of HRR as an indicator of scuba diver stress for underwater human-robot interaction (UHRI) scenarios is hitherto unexplored. In this paper, we introduce a novel approach for robotic estimation of HRR from underwater visual data by observing the volume of bubbles from exhalation states in scuba diving to measure the respiration rate. We propose and exploit a fuzzy labeling system that uses audio information to annotate exhalation and inhalation states based on the presence of bubbles. We collect an expansive audio and visual dataset of diver breathing from diverse geographical locations and water bodies, on which we compare four different methods for classifying the presence of bubbles in images. We then use a novel estimation algorithm to assess the diver respiration state from visual classification for computing the HRR in breaths per minute. Ultimately, we demonstrate the efficacy of our method in estimating HRR by comparing the respiration rate output against those measured by human analysts.
AlivertiA (2017) Wearable technology: role in respiratory health and disease. Breathe13(2): e27–e36.
3.
AlwasselHMahajanDKorbarB, et al. (2020) Self-supervised learning by cross-modal audio-video clustering In: Proceedings of the Conference on Neural Information Processing Systems. NeurIPS, Vol. 33, 9758–9770.
4.
ArandjelovicRZissermanA (2017) Look, listen and learn In: Proceedings of the IEEE International Conference on Computer Vision. ICCV, 609–617.
5.
AufderheideTPLurieKG (2004) Death by hyperventilation: a common and life-threatening problem during cardiopulmonary resuscitation. Critical Care Medicine32: S345–S351.
6.
AzadRKazerouniAHeidariM, et al. (2024) Advances in medical image analysis with vision transformers: a comprehensive review. Medical Image Analysis91: 103000.
7.
BehroozpourBSandbornPAWuMC, et al. (2017) Lidar system architectures and circuits. IEEE Communications Magazine55(10): 135–142.
8.
BinghamBFoleyBSinghH, et al. (2010) Robotic tools for deep water archaeology: surveying an ancient shipwreck with an autonomous underwater vehicle. Journal of Field Robotics27(6): 702–717.
BricoutAFontecave-JallonJColasD, et al. (2019) Adaptive accelerometry derived respiration: Comparison with respiratory inductance plethysmography during sleep In: Proceedings of the International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC). IEEE, 6714–6717.
11.
ChiarellaDBibuliMBruzzoneG, et al. (2015) Gesture-based language for diver-robot underwater interaction In: Proceedings of the IEEE/MTS OCEANS. IEEE, 1–9.
12.
CongYGuCZhangT, et al. (2021) Underwater robot sensing technology: a survey. Fundamental Research1(3): 337–345.
13.
CretikosMABellomoRHillmanK, et al. (2008) Respiratory rate: the neglected vital sign. Medical Journal of Australia188(11): 657–659.
14.
DudekGGiguerePPrahacsC, et al. (2007) Aqua: an amphibious autonomous robot. Computer40(1): 46–53.
15.
EbrahimianSNahviATashakoriM, et al. (2022) Multi-level classification of driver drowsiness by simultaneous analysis of ECG and respiration signals using deep neural networks. International Journal of Environmental Research and Public Health19(17): 10736.
16.
EdgeCEnanSSFultonM, et al. (2020) Design and experiments with LoCO AUV: a low cost open-source autonomous underwater vehicle In: Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 1761–1768.
17.
FultonMHongJSattarJ (2022) Using monocular vision and human body priors for AUVs to autonomously approach divers In: Proceedings of the IEEE International Conference on Robotics and Automation (ICRA). IEEE, 1076–1082.
18.
FultonMPrabhuASattarJ (2023a) HREyes: design, development, and evaluation of a novel method for AUVs to communicate information and gaze directionProceedings of the IEEE International Conference on Robotics and Automation (ICRA). IEEE, 7468–7475.
19.
FultonMSattarJAbsarR (2023b) SIREN: underwater robot-to-human communication using audio. IEEE Robotics and Automation Letters8(10): 6139–6146.
GiannakakisGGrigoriadisDGiannakakiK, et al. (2019) Review on psychological stress detection using biosignals. IEEE Transactions on Affective Computing13(1): 440–460.
GolpeRJimeACarpizoR (2002) Home sleep studies in the assessment of sleep apnea/hypopnea syndrome. Chest122(4): 1156–1161.
24.
GoodfellowIBengioYCourvilleA, et al. (2016) Deep Learning. MIT press Cambridge, Vol. 1.
25.
GrossmanP (1983) Respiration, stress, and cardiovascular function. Psychophysiology20(3): 284–300.
26.
HochreiterSSchmidhuberJ (1997) Long short-term memory. Neural Computation9(8): 1735–1780.
27.
HongWLaoJRenW, et al. (2022) Training object detectors from scratch: an empirical study in the era of vision transformer In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 4662–4671.
28.
HongJEnanSSSattarJ (2024) Diver identification using anthropometric data ratios for underwater multi-human-robot collaboration. IEEE Robotics and Automation Letters.
29.
IslamMJHoMSattarJ (2019) Understanding human motion and gestures for underwater human-robot collaboration. Journal of Field Robotics36(5): 851–873.
30.
IslamMJXiaYSattarJ (2020) Fast underwater image enhancement for improved visual perception. IEEE Robotics and Automation Letters5(2): 3227–3234.
31.
JacksonJD (1999) Classical electrodynamics.
32.
JasperAgbakwuru (2012) Oil/gas pipeline leak inspection and repair in underwater poor visibility conditions: challenges and perspectives. Journal of Environmental Protection3: 394–399. doi: 10.4236/jep.2012.35049
33.
JeongJJangYLeeI, et al. (2009) Wearable respiratory rate monitoring using piezo-resistive fabric sensor In: Proceedings of the World Congress on Medical Physics and Biomedical Engineering. Springer, 282–284.
34.
JiangHZhangW (2025) “Assistant or Replacement?” A socio-technical exploration of robots in underwater archaeology In: Proceedings of the ACM/IEEE International Conference on Human-Robot Interaction, 132–141.
35.
KhanSNaseerMHayatM, et al. (2022) Transformers in vision: a survey. ACM Computing Surveys54(10s): 1–41.
36.
KingmaDPBaJ (2014) Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.69806: 1412.
KrippendorffK (2018) Content Analysis: An Introduction to its Methodology. Sage publications.
39.
KutzkeDTWariarASattarJ (2024) Autonomous robotic re-alignment for face-to-face underwater human-robot interaction In: Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), 5920–5926.
40.
LapierreLSoetantoD (2007) Nonlinear path-following control of an AUV. Ocean Engineering34(11–12): 1734–1744.
41.
LiAQCoskunADohertyS, et al. (2016) Vision-based shipwreck mapping: on evaluating features quality and open source state estimation packages In: Proceedings of the IEEE/MTS OCEANS. IEEE, 1–10.
42.
MaclinC (1983) An overview of diving, salvage, offshore pollution abatement, and underwater ship maintenance. In: Proceedings of the IEEE/MTS OCEANS. IEEE, 391–393.
43.
ManningCD (2008) Introduction to Information Retrieval. Syngress Publishing.
44.
MarlingeMCoulangeMFitzpatrickRC, et al. (2019) Physiological stress markers during breath-hold diving and SCUBA diving. Physiological Reports7(6): e14033.
45.
MeyersD (2000) Safe diving operations on near-shore coastal construction projects In: Proceedings of the International Symposium on Underwater Technology (Cat. No. 00EX418). IEEE, 499–502.
46.
ModasshirMRahmanSYoungquistO, et al. (2018) Coral identification and counting with an autonomous underwater vehicle In: Proceedings of the IEEE International Conference on Robotics and Biomimetics (ROBIO). IEEE, 524–529.
47.
NesarMSBTrippeKStapleyR, et al. (2022) Improving touchless respiratory monitoring via lidar orientation and thermal imaging In: Proceedings of the IEEE Aerospace Conference (AERO). IEEE, 1–8.
48.
PaullLSaeediSSetoM, et al. (2013) AUV navigation and localization: a review. IEEE Journal of Oceanic Engineering39(1): 131–149.
49.
PerrySKhovanovaNAKhovanovIA (2019) Control of heart rate through guided high-rate breathing. Scientific Reports9(1): 1545.
50.
PetillotYRReedSRBellJM (2002) Real time AUV pipeline detection and tracking using side scan sonar and multi-beam echo-sounder In: Proceedings of the IEEE/MTS OCEANS. IEEE, Vol. 1, 217–222.
51.
PribanI (1963) An analysis of some short-term patterns of breathing in man at rest. The Journal of Physiology166(3): 425–434.
RoppoloLPWestfallAPepePE, et al. (2009) Dispatcher assessments for agonal breathing to improve detection of cardiac arrest. Resuscitation80(7): 769–772.
54.
SanyalSNundyKK (2018) Algorithms for monitoring heart rate and respiratory rate from the video of a user’s face. IEEE Journal of Translational Engineering in Health and Medicine6: 1–11.
55.
SchäferAKratkyKW (2008) Estimation of breathing rate from respiratory sinus arrhythmia: Comparison of various methods. Annals of Biomedical Engineering36: 476–485.
56.
SchwerdtfegerARSchwarzGPfurtschellerK, et al. (2020) Heart rate variability (HRV): from brain death to resonance breathing at 6 breaths per minute. Clinical Neurophysiology: Official Journal of the International Federation of Clinical Neurophysiology131(3): 676–693.
57.
SmithSW (1997) The Scientist and Engineer’s Guide to Digital Signal Processing. San Diego, CA: California Technical Publishing.
SuessWMAlexanderABSmithDD, et al. (1980) The effects of psychological stress on respiration: a preliminary study of anxiety and hyperventilation. Psychophysiology17(6): 535–540.
60.
TiptonMEglinCGennserM, et al. (1999) Immersion deaths and deterioration in swimming performance in cold water. Lancet (London, England)354(9179): 626–629.
61.
Van WaartHHarrisRJGantN, et al. (2020) Deep anaesthesia: the Thailand cave rescue and its implications for management of the unconscious diver underwater. Diving and Hyperbaric Medicine50(2): 121–129.
62.
VaswaniA (2017) Attention is all you need. Advances in Neural Information Processing Systems.
63.
VilasMGSchaumlöffelTRoigG (2024) Analyzing vision transformers for image classification in class embedding space. Advances in Neural Information Processing Systems36: 40030–40041.
64.
VuMTLe ThanhHNNHuynhTT, et al. (2021) Station-keeping control of a hovering over-actuated autonomous underwater vehicle under ocean current effects and model uncertainties in horizontal plane. IEEE Access9: 6855–6867.
65.
WangHZhangS (2023) Non-contact human respiratory rate measurement under dark environments by low-light video enhancement. Biomedical Signal Processing and Control85: 104874.
66.
YuanLChenYWangT, et al. (2021) Tokens-to-Token ViT: training vision transformers from scratch on ImageNetProceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 558–567.
67.
YuhJ (2000) Design and control of autonomous underwater robots: a survey. Autonomous Robots8: 7–24.
68.
ZamanKSahMDirekogluC, et al. (2023) A survey of audio classification using deep learning. IEEE Access.
69.
ZürnJBurgardWValadaA (2021) Self-supervised visual terrain classification from unsupervised acoustic feature learning. IEEE Transactions on Robotics37(2): 466–481.
