Recent advances in non-invasive physiologic monitoring leverage machine learning to provide unobtrusive, real-time assessments of a learner’s cognitive load (CL) as they engage in specific tasks. However, the performance characteristics of these novel composite physiologic CL measures are incompletely understood.
Objectives
We aimed to 1) explore the feasibility of measuring CL in real time using physiologically-derived inputs; 2) evaluate the performance characteristics of a novel composite CL measure during simulated virtual reality resuscitations; and 3) understand how this measure compares to traditional, self-reported measures of CL.
Methods
Novice (PGY1-2 pediatric residents) and expert (pediatric emergency medicine fellows and attendings) participants completed four virtual reality simulations as team leader. The scenario content (status epilepticus versus anaphylaxis) and level of distraction (high versus low) were manipulated. Cognitive load was measured in all participants using electroencephalography and electrocardiography data (“real-time CL”) as well as through self-report (NASA-TLX). Scenario performance also was measured.
Results
Complete data were available for 6 experts and 6 novices. Experts generally had lower CL than novices on both measures. Both measures localized the most significant differences between groups to the anaphylaxis scenarios (real-time CL [low-distraction] Cohen’s d -1.33 [95% CI -.2.56, -0.03] and self-reported CL [high-distraction] Cohen’s d -1.41 [95% CI -2.67, -0.10]). No consistent differences were seen with respect to level of distraction. Performance was similar between the two groups, though both exhibited fewer errors over time (F(3,48) = 5.75, p = .002).
Conclusion
It is feasible to unobtrusively measure cognitive load in real time during virtual reality simulations. There was convergence between the two CL measures: in both, experts had lower CL than novices, with the most significant effect size differences in the more challenging anaphylaxis scenarios.
AldekhylS.CavalcantiR. B.NaismithL. M. (2018). Cognitive load predicts point-of-care ultrasound simulator performance. Perspect Med Educ, 7(1), 23-32. https://doi.org/10.1007/s40037-017-0392-7
2.
AyoubA.PulijalaY. (2019). The application of virtual reality and augmented reality in Oral & Maxillofacial Surgery. BMC Oral Health, 19(1), 238. https://doi.org/10.1186/s12903-019-0937-8
3.
AyresP.LeeJ. Y.PaasF.van MerriënboerJ. J. G. (2021). The Validity of Physiological Measures to Identify Differences in Intrinsic Cognitive Load. Front Psychol, 12, 702538. https://doi.org/10.3389/fpsyg.2021.702538
4.
BarteitS.LanfermannL.BärnighausenT.NeuhannF.BeiersmannC. (2021). Augmented, Mixed, and Virtual Reality-Based Head-Mounted Devices for Medical Education: Systematic Review. JMIR Serious Games, 9(3), e29080. https://doi.org/10.2196/29080
5.
BeattieK. L.HillA.HorswillM. S.GroveP. M.StevensonA. R. L. (2021). Laparoscopic skills training: the effects of viewing mode (2D vs. 3D) on skill acquisition and transfer. Surg Endosc, 35(8), 4332-4344. https://doi.org/10.1007/s00464-020-07923-8
6.
BeaubienJ. M.WigginsS. L.DePriestW. N. (2019). Real-Time Measurement of Team Cognitive Load During Simulation-Based Training. In Paper presented at the Interservice/Industry Training, Simulation, and Education Conference. Orlando, FL, 2019.
7.
BrondfieldS.BlumA. M.LeeK.LinnM. C.O’SullivanP. S. (2021). The cognitive load of inpatient consults: development of the consult cognitive load instrument and initial validity evidence. Academic Medicine, 96(12), 1732-1741.
8.
ChangT. P.BeshayY.HollingerT.ShermanJ. M. (2019). Comparisons of Stress Physiology of Providers in Real-Life Resuscitations and Virtual Reality-Simulated Resuscitations. Simul Healthc, 14(2), 104-112. https://doi.org/10.1097/sih.0000000000000356
9.
ChangT. P.HollingerT.DolbyT.ShermanJ. M. (2021). Development and Considerations for Virtual Reality Simulations for Resuscitation Training and Stress Inoculation. Simul Healthc, 16(6), e219-e226. https://doi.org/10.1097/sih.0000000000000521
10.
ChenF. Q.LengY. F.GeJ. F.WangD. W.LiC.ChenB.SunZ. L. (2020). Effectiveness of Virtual Reality in Nursing Education: Meta-Analysis. J Med Internet Res, 22(9), e18290. https://doi.org/10.2196/18290
11.
CohenJ. (1988). Statistical Power Analysis for the Behavioral Sciences. New York, NY: Routledge Academic.
12.
CranfordK. N.TiettmeyerJ. M.ChuprinkoB. C.JordanS.GroveN. P. (2014). Measuring load on working memory: the use of heart rate as a means of measuring chemistry students’ cognitive load. Journal of Chemical Education, 91(5), 641-647.
13.
DanA.ReinerM. (2017). Real time EEG based measurements of cognitive load indicates mental states during learning. Journal of Educational Data Mining, 9(2), 31-44.
14.
DankbaarM.E.W.AlsmaJ.JansenE.E.H.van MerrienboerJ.J.G.van SaaseJ.L.C.MSchuitS.C.E.An experimental study on the effects of a simulation game on students' clinical cognitive skills and motivation. Adv Health Sci Educ Theory Pract,2016; 21(3): 505-521.
15.
DiasR. D.ConboyH. M.GabanyJ. M.ClarkeL. A.OsterweilL. J.AvruninG. S.ZenatiM. A. (2018a). Development of an Interactive Dashboard to Analyze Cognitive Workload of Surgical Teams During Complex Procedural Care. IEEE Int Interdiscip Conf Cogn Methods Situat Aware Decis Support, 2018, 77-82. https://doi.org/10.1109/cogsima.2018.8423995
16.
DiasR. D.Ngo-HowardM. C.BoskovskiM. T.ZenatiM. A.YuleS. J. (2018b). Systematic review of measurement tools to assess surgeons' intraoperative cognitive workload. Journal of British Surgery, 105(5), 491-501.
17.
DurkeeK.GeyerA.PappadaS.OrtizA.GalsterS. (2013). Real-time workload assessment as a foundation for human performance augmentation. In Paper presented at the International Conference on Augmented Cognition.Las Vegas, NV, 2013.
18.
Esteban-GuitartM. (2018). The biosocial foundation of the early Vygotsky: Educational psychology before the zone of proximal development. History of psychology, 21(4), 384.
19.
FontaineG.CossetteS.Maheu-CadotteM. A.MailhotT.DeschênesM. F.Mathieu-DupuisG.DubéV. (2019). Efficacy of adaptive e-learning for health professionals and students: a systematic review and meta-analysis. BMJ Open, 9(8), e025252. https://doi.org/10.1136/bmjopen-2018-025252
FrederiksenJ. G.SørensenS. M. D.KongeL.SvendsenM. B. S.Nobel-JørgensenM.BjerrumF.AndersenS. A. W. (2020). Cognitive load and performance in immersive virtual reality versus conventional virtual reality simulation training of laparoscopic surgery: a randomized trial. Surg Endosc, 34(3), 1244-1252. https://doi.org/10.1007/s00464-019-06887-8
22.
GentiliR. J.RietschelJ. C.JaquessK. J.LoL.-C.PrevostC. M.MillerM. W.HatfieldB. D. (2014). Brain biomarkers based assessment of cognitive workload in pilots under various task demands. In Paper presented at the 2014 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society. Chicago, IL, 2014.
23.
GerjetsP.WalterC.RosenstielW.BogdanM.ZanderT. O. (2014). Cognitive state monitoring and the design of adaptive instruction in digital environments: lessons learned from cognitive workload assessment using a passive brain-computer interface approach. Front Neurosci, 8, 385. https://doi.org/10.3389/fnins.2014.00385
24.
GrootF.JonkerG.RiniaM.Ten CateO.HoffR. G. (2020). Simulation at the frontier of the zone of proximal development: A test in acute care for inexperienced learners. Academic Medicine, 95(7), 1098-1105.
25.
HajiF. A.CheungJ. J.WoodsN.RegehrG.de RibaupierreS.DubrowskiA. (2016). Thrive or overload? The effect of task complexity on novices’ simulation‐based learning. Medical education, 50(9), 955-968.
26.
HartS. G. (2006). NASA-task load index (NASA-TLX); 20 years later. In Paper presented at the Proceedings of the human factors and ergonomics society annual meeting.San Francisco, CA, 2006.
27.
HartS. G.StavelandL. E. (1988). Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research. In Advances in psychology (Vol. 52, pp. 139-183): Elsevier.
28.
HockeyG. R. (1997). Compensatory control in the regulation of human performance under stress and high workload; a cognitive-energetical framework. Biol Psychol, 45(1-3), 73-93. https://doi.org/10.1016/s0301-0511(96)05223-4
29.
KellmanP. J.KrasneS. (2018). Accelerating expertise: Perceptual and adaptive learning technology in medical learning. Med Teach, 40(8), 797-802. https://doi.org/10.1080/0142159x.2018.1484897
30.
KrasneS.StevensC. D.KellmanP. J.NiemannJ. T. (2021). Mastering Electrocardiogram Interpretation Skills Through a Perceptual and Adaptive Learning Module. AEM Educ Train, 5(2), e10454. https://doi.org/10.1002/aet2.10454
31.
LapierreA.ArbourC.Maheu-CadotteM.-A.VinetteB.FontaineG.LavoieP. (2022). Association between Clinical Simulation Design Features and Novice Healthcare Professionals’ Cognitive Load: A Systematic Review and Meta-Analysis. Simulation & Gaming, 53(5), 538-563. https://doi.org/10.1177/10468781221120599
32.
LinY.WangS. (2019). The study and application of adaptive learning method based on virtual reality for engineering education. In Paper presented at the International Conference on Image and Graphics.Beijing, China, 2019.
33.
LoweD. J.JamesS. A.LloydA.CleggG. R. (2016). Feasibility of EEG to monitor cognitive performance during venous cannulation: EEG Distracted Intravenous Access (E-DIVA). BMJ Simul Technol Enhanc Learn, 2(3), 68-72. https://doi.org/10.1136/bmjstel-2015-000082
McGrathJ. L.TaekmanJ. M.DevP.DanforthD. R.MohanD.KmanN.BondW. F. (2018). Using Virtual Reality Simulation Environments to Assess Competence for Emergency Medicine Learners. Acad Emerg Med, 25(2), 186-195. https://doi.org/10.1111/acem.13308
36.
MorraM.BraundH.HallA. K.SzulewskiA. (2021). Cognitive load and processes during chest radiograph interpretation in the emergency department across the spectrum of expertise. AEM Educ Train, 5(4), e10693. https://doi.org/10.1002/aet2.10693
37.
NaismithL. M.CavalcantiR. B. (2015). Validity of Cognitive Load Measures in Simulation-Based Training: A Systematic Review. Acad Med, 90(11 Suppl), S24-35. https://doi.org/10.1097/acm.0000000000000893
38.
NikoulineA.JimenezM. C.OkrainecA. (2020). Feasibility of remote administration of the fundamentals of laparoscopic surgery (FLS) skills test using Google wearable device. Surg Endosc, 34(1), 443-449. https://doi.org/10.1007/s00464-019-06788-w
39.
OsadchyiV.ChemerysH.OsadchaK.KruhlykV. S.KoniukhovS.KivA. (2020). Conceptual model of learning based on the combined capabilities of augmented and virtual reality technologies with adaptive learning systems. In Paper presented at the CEUR Workshop Proceedings.Kryvyi Rih, Ukraine, 2020.
40.
PaasF.RenklA.SwellerJ. (2003). Cognitive load theory and instructional design: Recent developments. Educational psychologist, 38(1), 1-4.
41.
PaasF. G. (1992). Training strategies for attaining transfer of problem-solving skill in statistics: a cognitive-load approach. Journal of educational psychology, 84(4), 429.
42.
PappadaS.PapadimosT.LippsJ.FeeneyJ.DurkeeK.GalsterS.BhandaryS. (2016). Establishing an instrumented environment for simulation-based training of healthcare providers: an initial proof of concept. International Journal of Academic Medicine, 124.
43.
PollackA. H.PrattW. (2020). Association of Health Record Visualizations With Physicians' Cognitive Load When Prioritizing Hospitalized Patients. JAMA Netw Open, 3(1), e1919301. https://doi.org/10.1001/jamanetworkopen.2019.19301
44.
RölfingJ. D.NørskovJ. K.PaltvedC.KongeL.AndersenS.A.W.Failure affects subjective estimates of cognitive load through a negative carry-over effect in virtual reality simulation of hip fracture surgery. Adv Simul (Lond); 2019; 4: 26.
45.
RomineW. L.SchroederN. L.GraftJ.YangF.SadeghiR.ZabihimayvanM.BanerjeeT. (2020). Using Machine Learning to Train a Wearable Device for Measuring Students' Cognitive Load during Problem-Solving Activities Based on Electrodermal Activity, Body Temperature, and Heart Rate: Development of a Cognitive Load Tracker for Both Personal and Classroom Use. Sensors (Basel), 20(17), 17. https://doi.org/10.3390/s20174833
46.
RomitoB. T.KrasneS.KellmanP. J.DhillonA. (2016). The impact of a perceptual and adaptive learning module on transoesophageal echocardiography interpretation by anaesthesiology residents. Br J Anaesth, 117(4), 477-481. https://doi.org/10.1093/bja/aew295
47.
RubertoA. J.RodenburgD.RossK.SarkarP.HunglerP. C.EtemadA.SzulewskiA. (2021). The future of simulation-based medical education: Adaptive simulation utilizing a deep multitask neural network. AEM Educ Train, 5(3), e10605. https://doi.org/10.1002/aet2.10605
48.
SewellJ.L.BowenJ.L.ten CateO.O'SullivanP.S.ShahB.BoscardinC.K.Learning challenges, teaching strategies, and cognitive load: insights from the experience of seasoned endoscopy teachers. Academic Medicine, 2020; 95(5): 794-802.
49.
SewellJ. L.MaggioL. A.Ten CateO.van GogT.YoungJ. Q.O’SullivanP. S. (2019). Cognitive load theory for training health professionals in the workplace: A BEME review of studies among diverse professions: BEME Guide No. 53. Medical teacher, 41(3), 256-270.
SiegelE.WeiJ.GomesA.OlivieraM.SundaramoorthyP.SmathersK.BailensonJ. (2021). HP Omnicept Cognitive Load Database (HPO-CLD)–Developing a Multimodal Inference Engine for Detecting Real-time Mental Workload in VR. Retrieved from
52.
SolhjooS.HaigneyM. C.McBeeE.van MerrienboerJ. J. G.SchuwirthL.ArtinoA. R.Jr.DurningS. J. (2019). Heart Rate and Heart Rate Variability Correlate with Clinical Reasoning Performance and Self-Reported Measures of Cognitive Load. Sci Rep, 9(1), 14668. https://doi.org/10.1038/s41598-019-50280-3
SzulewskiA.GegenfurtnerA.HowesD. W.SivilottiM. L. A.van MerriënboerJ. J. G. (2017). Measuring physician cognitive load: validity evidence for a physiologic and a psychometric tool. Adv Health Sci Educ Theory Pract, 22(4), 951-968. https://doi.org/10.1007/s10459-016-9725-2
55.
SzulewskiA.HowesD.van MerriënboerJ. J. G.SwellerJ. (2021). From Theory to Practice: The Application of Cognitive Load Theory to the Practice of Medicine. Acad Med, 96(1), 24-30. https://doi.org/10.1097/acm.0000000000003524
56.
SzulewskiA.RothN.HowesD. (2015). The Use of Task-Evoked Pupillary Response as an Objective Measure of Cognitive Load in Novices and Trained Physicians: A New Tool for the Assessment of Expertise. Acad Med, 90(7), 981-987. https://doi.org/10.1097/acm.0000000000000677
57.
Van GogT.PaasF. (2011). Cognitive Load Measurement. In SeelN. M. (Ed.), Encyclopedia of the Sciences of Learning (pp. 599-601). New York, NY: Springer Science & Business Media.
58.
van MerriënboerJ. J.SwellerJ. (2010). Cognitive load theory in health professional education: design principles and strategies. Med Educ, 44(1), 85-93. https://doi.org/10.1111/j.1365-2923.2009.03498.x
59.
VegaA. V.MadrigalO. C.KugurakovaV. (2021). Approach of immersive adaptive learning for Virtual Reality simulator. In Paper presented at the Anais do III Workshop on Advanced Virtual Environments and Education.Fortaleza, CE, Brazil, 2021.
60.
VygotskyL. S.ColeM. (1978). Mind in society: Development of higher psychological processes: Harvard university press.
61.
XiongR.KongF.YangX.LiuG.WenW. (2020). Pattern Recognition of Cognitive Load Using EEG and ECG Signals. Sensors (Basel), 20(18). https://doi.org/10.3390/s20185122
62.
YoungJ. Q.BoscardinC. K.van DijkS. M.AbdullahR.IrbyD. M.SewellJ. L.O’SullivanP. S. (2016). Performance of a cognitive load inventory during simulated handoffs: evidence for validity. Sage Open Medicine, 4, 2050312116682254.
63.
YoungJ. Q.Van MerrienboerJ.DurningS.Ten CateO. (2014). Cognitive Load Theory: implications for medical education: AMEE Guide No. 86. Med Teach, 36(5), 371-384. https://doi.org/10.3109/0142159x.2014.889290
64.
YurkoY. Y.ScerboM. W.PrabhuA. S.AckerC. E.StefanidisD. (2010). Higher mental workload is associated with poorer laparoscopic performance as measured by the NASA-TLX tool. Simul Healthc, 5(5), 267-271. https://doi.org/10.1097/SIH.0b013e3181e3f329
65.
ZakeriZ.MansfieldN.SunderlandC.OmurtagA. (2020). Physiological correlates of cognitive load in laparoscopic surgery. Sci Rep, 10(1), 12927. https://doi.org/10.1038/s41598-020-69553-3
66.
ZakharyB. M.KamL. M.KaufmanB. S.FelnerK. J. (2017). The Utility of High-Fidelity Simulation for Training Critical Care Fellows in the Management of Extracorporeal Membrane Oxygenation Emergencies: A Randomized Controlled Trial. Crit Care Med, 45(8), 1367-1373. https://doi.org/10.1097/ccm.0000000000002437
67.
ZhengW.-L.ZhuJ.-Y.LuB.-L. (2017). Identifying stable patterns over time for emotion recognition from EEG. IEEE Transactions on Affective Computing, 10(3), 417-429.
68.
ZuT.MunsellJ.RebelloN.S.Subjective measure of cognitive load depends on participants' content knowledge level. Frontiers in Education; 2021; 6: 647097. https://doi.org/10.3389/feduc.2021.647097
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
