An Adaptive Human-Robotic Interaction Architecture for Augmenting Surgery Performance Using Real-Time Workload Sensing—Demonstration of a Semi-autonomous Suction Tool

Abstract

Objective

This study developed and evaluated a mental workload-based adaptive automation (MWL-AA) that monitors surgeon cognitive load and assist during cognitively demanding tasks and assists surgeons in robotic-assisted surgery (RAS).

Background

The introduction of RAS makes operators overwhelmed. The need for precise, continuous assessment of human mental workload (MWL) states is important to identify when the interventions should be delivered to moderate operators’ MWL.

Method

The MWL-AA presented in this study was a semi-autonomous suction tool. The first experiment recruited ten participants to perform surgical tasks under different MWL levels. The physiological responses were captured and used to develop a real-time multi-sensing model for MWL detection. The second experiment evaluated the effectiveness of the MWL-AA, where nine brand-new surgical trainees performed the surgical task with and without the MWL-AA. Mixed effect models were used to compare task performance, objective- and subjective-measured MWL.

Results

The proposed system predicted high MWL hemorrhage conditions with an accuracy of 77.9%. For the MWL-AA evaluation, the surgeons’ gaze behaviors and brain activities suggested lower perceived MWL with MWL-AA than without. This was further supported by lower self-reported MWL and better task performance in the task condition with MWL-AA.

Conclusion

A MWL-AA systems can reduce surgeons' workload and improve performance in a high-stress hemorrhaging scenario. Findings highlight the potential of utilizing MWL-AA to enhance the collaboration between the autonomous system and surgeons. Developing a robust and personalized MWL-AA is the first step that can be used do develop additional use cases in future studies.

Application

The proposed framework can be expanded and applied to more complex environments to improve human-robot collaboration.

Keywords

physiological measurement mental workload robotic and telesurgery artificial intelligence adaptive automation

Get full access to this article

View all access options for this article.

References

Aricò

Borghini

di Flumeri

Colosimo

Bonelli

Golfetti

Pozzi

Imbert

J. P.

Granger

Benhacene

Babiloni

(2016). Adaptive automation triggered by EEG-based mental workload index: A passive brain-computer interface application in realistic air traffic control environment. Frontiers in Human Neuroscience, 10(OCT2016), 1–13. https://doi.org/10.3389/fnhum.2016.00539

Bates

et al. (2015). Fitting linear mixed-effects models using lme4. In Journal of Statistical Software (1st ed., 67, pp. 27–48). Foundation for Open Access Statistics.

Bigdely-Shamlo

Mullen

Kothe

K. M.

Robbins

K. A.

(2015). The PREP pipeline: Standardized preprocessing for large-scale EEG analysis. Frontiers in Neuroinformatics, 9(JUNE), 1–19. https://doi.org/10.3389/fninf.2015.00016

Byrne

E. A.

Parasuraman

(1996). Psychophysiology and adaptive automation. Biological Psychology, 42(3), 249–268. https://doi.org/10.1016/0301-0511(95)05161-9

Czarnik

R. E.

Stone

K. S.

Everhart

C. C.

Preusser

B. A.

(1991). Differential effects of continuous versus intermittent suction on tracheal tissue. Heart & Lung: The Journal of Critical Care, 20(2), 144–151, http://europepmc.org/abstract/MED/2004925

de Visser

E. J.

Legoullon

Freedy

Weltman

Parasuraman

(2008). Designing an adaptive automation system for human supervision of unmanned vehicles: A bridge from theory to practice. Proceedings of the Human Factors and Ergonomics Society, 52(4), 221–225. https://doi.org/10.1177/154193120805200405

Debie

Fernandez Rojas

Fidock

Barlow

Kasmarik

Anavatti

Garratt

Abbass

H. A.

(2021). Multimodal fusion for objective assessment of cognitive workload: A review. IEEE Transactions on Cybernetics, 51(3), 1542–1555. https://doi.org/10.1109/TCYB.2019.2939399

Delorme

Makeig

(2004). EEGLAB: An open source toolbox for analysis of single-trial EEG dynamics. Journal of Neuroscience Methods, 134(1), 9–21.

Escalona

E. M.

et al.. (2019). Latency Differences Between Mental Workload Measures in Detecting Workload Changes. Communications in Computer and Information Science, 1012(1), 131–146. DOI:10.1007/978-3-030-14273-5_8

10.

Fernandez Rojas

Debie

Fidock

Barlow

Kasmarik

Anavatti

Garratt

Abbass

(2020). Electroencephalographic workload indicators during teleoperation of an unmanned aerial vehicle shepherding a swarm of unmanned ground vehicles in contested environments. Frontiers in Neuroscience, 0, 40. https://doi.org/10.3389/FNINS.2020.00040

11.

Fidopiastis

C. M

(2009). Impact of Automation and Task Load on Unmanned System Operator’s Eye Movement Patterns. In Foundations of Augmented Cognition. Neuroergonomics and Operational Neuroscience (5638, pp. 229–238). Springer, Berlin, Heidelberg.

12.

Garreta

et al. (2013). Supervised Learning. In Learning scikit-learn: Machine Learning in Python Experience (12). Packt Publishing Ltd.

13.

Gevins

Smith

M. E.

Leong

McEvoy

Whitfield

Rush

(2016). Monitoring Working Memory Load during Computer-Based Tasks with EEG Pattern Recognition Methods: Human Factors: The Journal of the Human Factors and Ergonomics Society, 40(1), 79–91. https://doi.org/10.1518/001872098779480578

14.

Giedelman

C. A.

Abdul-Muhsin

Schatloff

Palmer

Lee

Sanchez-Salas

Cathelineau

Dávila

Cavelier

Rueda

Patel

(2013). The impact of robotic surgery in urology. Actas Urologicas Espanolas, 37(10), 652–657. https://doi.org/10.1016/j.acuro.2012.11.015

15.

Hafed

Z. M.

Krauzlis

R. J.

(2012). Similarity of superior colliculus involvement in microsaccade and saccade generation. Journal of Neurophysiology, 107(7), 1904–1916. https://doi.org/10.1152/jn.01125.2011

16.

Haidegger

(2019). Autonomy for surgical robots: Concepts and paradigms. IEEE Transactions on Medical Robotics and Bionics, 1(2), 65–76. https://doi.org/10.1109/tmrb.2019.2913282

17.

Hancock

P. A.

Matthews

(2019). Workload and performance: Associations, insensitivities, and dissociations. Human Factors, 61(3), 374–392. https://doi.org/10.1177/0018720818809590

18.

Hasselberg

Söffker

(2013). A human cognitive performance measure based on available options for adaptive aiding. IFAC Proceedings Volumes, 46(15), 442–449. https://doi.org/10.3182/20130811-5-US-2037.00012

19.

Hayden

R. W.

et al. (2012). A Review of: “An R Companion to Applied Regression, Second Edition, by J. Fox and S. Weisberg”. Journal of Biopharmaceutical Statistics, 22(2), 418–419. DOI:10.1080/10543406.2012.635980

20.

Helton

W. S.

Shaw

Warm

J. S.

Matthews

Hancock

(2008). Effects of warned and unwarned demand transitions on vigilance performance and stress. Anxiety, Stress and Coping, 21(2), 173–184. https://doi.org/10.1080/10615800801911305

21.

Hussein

Palangi

Ward

R. K.

Wang

Z. J.

(2019). Optimized deep neural network architecture for robust detection of epileptic seizures using EEG signals. Clinical Neurophysiology, 130(1), 25–37. https://doi.org/10.1016/j.clinph.2018.10.010

22.

Imants

de Greef

(2014). Eye metrics for task-dependent automation. ACM International Conference Proceeding Series. https://doi.org/10.1145/2637248.2637274

23.

Jacob

R. J. K

et al. (2003). Eye tracking in human-computer interaction and usability research. In The Mind’s Eye: Cognitive and Applied Aspects of Eye Movement Research (pp. 573-605). Elsevier Science BV.

24.

Kappel

S. L.

Rank

M. L.

Toft

H. O.

Andersen

Kidmose

(2019). Dry-contact electrode ear-EEG. IEEE Transactions on Biomedical Engineering, 66(1), 150–158. https://doi.org/10.1109/TBME.2018.2835778

25.

Kazanzidesf

Chen

Deguet

Fischer

G. S.

Taylor

R. H.

Dimaio

S. P.

(2014). An open-source research kit for the da Vinci® Surgical System. Proceedings - IEEE International Conference on Robotics and Automation, 6434–6439. https://doi.org/10.1109/ICRA.2014.6907809

26.

Ketkar

et al. (2017). Introduction to Keras. In Deep Learning with Python (pp. 97-111). Springer.

27.

Kirchner

E. A.

Kim

S. K.

Tabie

Wöhrle

Maurus

Kirchner

(2016). An intelligent man-machine interface—Multi-robot control adapted for task engagement based on single-trial detectability of P300. Frontiers in Human Neuroscience, (2016 Jun 2), 291. https://doi.org/10.3389/FNHUM.2016.00291

28.

Knaepen

Marusic

Crea

Rodríguez Guerrero

C. D.

Vitiello

Pattyn

Mairesse

Lefeber

Meeusen

(2015). Psychophysiological response to cognitive workload during symmetrical, asymmetrical and dual-task walking. Human Movement Science, 40, 248–263. https://doi.org/10.1016/j.humov.2015.01.001

29.

Kohlmorgen

Dornhege

Braun

Blankertz

Müller

K.-R.

Curio

Hagemann

Bruns

Schrauf

Kincses

(2007). Improving human performance in a real operating environment through real-time mental workload detection.

30.

Kondacs

Szabó

(1999). Long-term intra-individual variability of the background EEG in normals. Clinical Neurophysiology, 110(10), 1708–1716. https://doi.org/10.1016/S1388-2457(99)00122-4

31.

Kranzfelder

Staub

Fiolka

Schneider

Gillen

Wilhelm

Friess

Knoll

Feussner

(2013). Toward increased autonomy in the surgical OR: Needs, requests, and expectations. Surgical Endoscopy, 27(5), 1681–1688. https://doi.org/10.1007/s00464-012-2656-y

32.

Maggi

di Nocera

(2021). Sensitivity of the spatial distribution of fixations to variations in the type of task demand and its relation to visual entropy. Frontiers in Human Neuroscience, 15, 257. https://doi.org/10.3389/FNHUM.2021.642535

33.

Malpani

Vedula

S. S.

Lin

H. C.

Hager

G. D.

Taylor

R. H.

(2020). Effect of real-time virtual reality-based teaching cues on learning needle passing for robot-assisted minimally invasive surgery: a randomized controlled trial. International Journal of Computer Assisted Radiology and Surgery, 15(7), 1187–1194. https://doi.org/10.1007/s11548-020-02156-5

34.

Mapelli

Özkurt

T. E.

(2019). Brain Oscillatory Correlates of Visual Short-Term Memory Errors. Frontiers in Human Neuroscience, 13(33), 1–15. https://doi.org/10.3389/FNHUM.2019.00033

35.

Matthews

Reinerman-Jones

L. E.

Barber

D. J.

Julian Abich

(2014). The psychometrics of mental workload: multiple measures are sensitive but divergent. Human Factors: The Journal of the Human Factors and Ergonomics Society, 57(1), 125–143. https://doi.org/10.1177/0018720814539505

36.

Nagy

T. D.

Haidegger

(2019). A DVRK-based framework for surgical subtask automation. Acta Polytechnica Hungarica, 16(8), 61–78. https://doi.org/10.12700/APH.16.8.2019.8.5

37.

Natarajan

Acharya

U. R.

Alias

Tiboleng

Puthusserypady

S. K.

(2004). Nonlinear analysis of EEG signals at different mental states. BioMedical Engineering Online, 3(1), 1–11. https://doi.org/10.1186/1475-925X-3-7

38.

Paszkiel

(2020). Data Analysis of Human Brain Activity Using MATLAB Environment with EEGLAB. In Analysis and Classification of EEG Signals for Brain–Computer Interfaces (852, pp. 33-39). Springer, Cham.

39.

Pontes

F. J.

Amorim

G. F.

Balestrassi

P. P.

Paiva

A. P.

Ferreira

J. R.

(2016). Design of experiments and focused grid search for neural network parameter optimization. Neurocomputing, 186(01), 22–34. https://doi.org/10.1016/j.neucom.2015.12.061

40.

Power

Rafii-Tari

Bergeles

Vitiello

Yang

G. Z.

(2015). A cooperative control framework for haptic guidance of bimanual surgical tasks based on Learning from Demonstration. Proceedings - IEEE International Conference on Robotics and Automation, 2015(June), 5330–5337. https://doi.org/10.1109/ICRA.2015.7139943

41.

Prinzel

L. J.

Freeman

F. G.

Scerbo

M. W.

Mikulka

P. J.

Pope

A. T.

(2000). A closed-loop system for examining psychophysiological measures for adaptive task allocation. International Journal of Aviation Psychology, 10(4), 393–410. https://doi.org/10.1207/S15327108IJAP1004_6

42.

Puma

Matton

Paubel

P. v.

Raufaste

É.

El-Yagoubi

(2018). Using theta and alpha band power to assess cognitive workload in multitasking environments. International Journal of Psychophysiology, 123, 111–120. https://doi.org/10.1016/J.IJPSYCHO.2017.10.004

43.

Reason

(2000). Human error: Models and management. BMJ, 320(7237), 768–770. https://doi.org/10.1136/BMJ.320.7237.768

44.

Reimer

Mehler

Wang

Coughlin

J. F.

(2010). The impact of systematic variation of cognitive demand on drivers’ visual attention across multiple age groups. Proceedings of the Human Factors and Ergonomics Society, 3(first 4), 2052–2056. https://doi.org/10.1518/107118110X12829370264321

45.

Rieger

Fenger

Neubert

Weippert

Kreuzfeld

Stoll

(2015). Psychophysical workload in the operating room: primary surgeon versus assistant. Surgical Endoscopy, 29(7), 1990–1998. https://doi.org/10.1007/s00464-014-3899-6

46.

Russell

et al. (2022). Package ‘emmeans’. In R topics documented (1st ed., 34, pp. 216-221). CRAN.

47.

Spitzer

Haegens

(2017). Beyond the status quo: A role for beta oscillations in endogenous content (re)activation. ENeuro, 4(4). https://doi.org/10.1523/ENEURO.0170-17.2017

48.

Sterman

M. B.

Mann

C. A.

(1995). Concepts and applications of EEG analysis in aviation performance evaluation. Biological Psychology, 40(1–2), 115–130. https://doi.org/10.1016/0301-0511(95)05101-5

49.

Teo

Matthews

Reinerman-Jones

Barber

(2020). Adaptive aiding with an individualized workload model based on psychophysiological measures. Human-Intelligent Systems Integration, 2(1–4), 1–15. https://doi.org/10.1007/s42454-019-00005-8

50.

Teo

Reinerman-Jones

Matthews

Szalma

Jentsch

Hancock

(2018). Enhancing the effectiveness of human-robot teaming with a closed-loop system. Applied Ergonomics, 67, 91–103. https://doi.org/10.1016/J.APERGO.2017.07.007

51.

Tobii Manual

AB.

(2020). Tobii Pro Glasses 3 – groundbreaking wearable eye tracker to analyze human behavior (pp. 3-5). Press Release. https://www.tobii.com/group/news-media/press-releases/2020/6/tobii-pro-glasses-3--groundbreaking-wearable-eye-tracker-to-analyze-human-behavior/

52.

Weber

Catchpole

Becker

A. J.

Schlenker

Weigl

(2018). Effects of flow disruptions on mental workload and surgical performance in robotic-assisted surgery. World Journal of Surgery, 42(11), 3599–3607. https://doi.org/10.1007/s00268-018-4689-4

53.

Weigl

Stefan

Abhari

Wucherer

Fallavollita

Lazarovici

Weidert

Euler

Catchpole

(2016). Intra-operative disruptions, surgeon’s mental workload, and technical performance in a full-scale simulated procedure. Surgical Endoscopy, 30(2), 559–566. https://doi.org/10.1007/s00464-015-4239-1

54.

Weiss

Khoshgoftaar

T. M.

Wang

D. D.

(2016). A Survey of Transfer Learning Journal of Big Data (Vol. 3). Springer International Publishing. https://doi.org/10.1186/s40537-016-0043-6

55.

Wilson

G. F.

Russell

C. A.

(2003). Real-time assessment of mental workload using psychophysiological measures and artificial neural networks. Human Factors, 45(4), 635–643. https://doi.org/10.1518/HFES.45.4.635.27088

56.

Cha

Sulek

Zhou

Sundaram

C. P.

Wachs

(2020). Eye-tracking metrics predict perceived workload in robotic surgical skills training. Human Factors, 62(8), 1365–1386. https://doi.org/10.1177/0018720819874544

57.

Yuksel

B. F.

Oleson

K. B.

Harrison

Peck

E. M.

Afergan

Piano

(2016). Learn piano with BACh: An adaptive learning interface that adjusts task difficulty based on brain state recommended citation. https://doi.org/10.1145/2858036.2858388

58.

Yurko

Y. Y.

Scerbo

M. W.

Prabhu

A. S.

Acker

C. E.

Stefanidis

(2010). Higher mental workload is associated with poorer laparoscopic performance as measured by the NASA-TLX tool. Simulation in Healthcare, 5(5), 267–271. https://doi.org/10.1097/SIH.0b013e3181e3f329

59.

Zhou

Cha

J. S.

Gonzalez

Wachs

J. P.

Sundaram

C. P.

(2020). Multimodal physiological signals for workload prediction in robot-assisted surgery. ACM Transactions on Human-Robot Interaction, 9(2), 1–26. https://doi.org/10.1145/3368589

60.

Zhou

Kilsgaard

Moradi

Kappel

S. L.

Kidmose

(2016). A wearable ear-EEG recording system based on dry-contact active electrodes. IEEE Symposium on VLSI Circuits, Digest of Technical Papers, 7–8 September 2016. https://doi.org/10.1109/VLSIC.2016.7573559