Determine the relationship between cognitive load (CL) and automation bias (AB).
Background:
Clinical decision support (CDS) for electronic prescribing can improve safety but introduces the risk of AB, where reliance on CDS replaces vigilance in information seeking and processing. We hypothesized high CL generated by high task complexity would increase AB errors.
Method:
One hundred twenty medical students prescribed medicines for clinical scenarios using a simulated e-prescribing system in a randomized controlled experiment. Quality of CDS (correct, incorrect, and no CDS) and task complexity (low and high) were varied. CL, omission errors (failure to detect prescribing errors), and commission errors (acceptance of false positive alerts) were measured.
Results:
Increasing complexity from low to high significantly increased CL, F(1, 118) = 71.6, p < .001. CDS reduced CL in high-complexity conditions compared to no CDS, F(2, 117) = 4.72, p = .015. Participants who made omission errors in incorrect and no CDS conditions exhibited lower CL compared to those who did not, F(1, 636.49) = 3.79, p = .023.
Conclusion:
Results challenge the notion that AB is triggered by increasing task complexity and associated increases in CL. Omission errors were associated with lower CL, suggesting errors may stem from an insufficient allocation of cognitive resources.
Application:
This is the first research to examine the relationship between CL and AB. Findings suggest designers and users of CDS systems need to be aware of the risks of AB. Interventions that increase user vigilance and engagement may be beneficial and deserve further investigation.
AmmenwerthE.Schnell-InderstP.MachanC.SiebertU. (2008). The effect of electronic prescribing on medication errors and adverse drug events: A systematic review. Journal of the American Medical Informatics Association, 15(5), 585–600. doi:10.1197/jamia.M2667
2.
Australian Medicines Handbook Pty Ltd. (2015, January). Australian medicines handbook 2015. Retrieved from http://amhonline.amh.net.au/
3.
AyresP.SwellerJ. (1990). Locus of difficulty in multistage mathematics problems. American Journal of Psychology, 103(2), 167–193. doi:10.2307/1423141
4.
AyresP. L. (2001). Systematic mathematical errors and cognitive load. Contemporary Educational Psychology, 26(2), 227–248. doi:10.1006/ceps.2000.1051
5.
BaddeleyA. (1992). Working memory. Science, 255(5044), 556–559.
6.
BagheriN.JamiesonG. A. (2004a). Considering subjective trust and monitoring behavior in assessing automation-induced “complacency.”Mahwah, NJ: Lawrence Erlbaum.
7.
BagheriN.JamiesonG. A. (2004b, October10–13). The impact of context-related reliability on automation failure detection and scanning behaviour. Paper presented at the IEEE International Conference on Systems, Man and Cybernetics (IEEE Cat. No. 04CH37583), The Hague, the Netherlands.
8.
BahnerJ.ElepfandtM. F.ManzeyD. (2008, September22–26). Misuse of diagnostic aids in process control: The effects of automation misses on complacency and automation bias. Paper presented at the Human Factors and Ergonomics Society annual meeting, New York, NY.
9.
BahnerJ.HuperA.-D.ManzeyD. (2008). Misuse of automated decision aids: Complacency, automation bias and the impact of training experience. International Journal of Human-Computer Studies, 66(9), 688–699. doi:10.1016/j.ijhcs.2008.06.001
10.
BaileyN. R.ScerboM. W. (2007). Automation-induced complacency for monitoring highly reliable systems: The role of task complexity, system experience, and operator trust. Theoretical Issues in Ergonomics Science, 8(4), 321–348. doi:10.1080/14639220500535301
11.
BaileyN. R.ScerboM. W.FreemanF. G.MikulkaP. J.ScottL. A. (2006). Comparison of a brain-based adaptive system and a manual adaptable system for invoking automation. Human Factors, 48, 693–709. doi:10.1518/001872006779166280
12.
BlandJ. M.AltmanD. G. (1995). Multiple significance tests: The Bonferroni method. BMJ, 310(6973), 170. doi:10.1136/bmj.310.6973.170
13.
BrittH.MillerG.HendersonJ.BayramC.HarrisonC.ValentiL.. . .CharlesJ. (2015). General practice activity in Australia 2014–15. General practice series no. 38. Sydney, Australia: Sydney University Press.
14.
CowanN. (2001). The magical number 4 in short-term memory: A reconsideration of mental storage capacity. Behavioral Brain Science, 24, 87–185.
15.
FiskeS. T.TaylorS. E. (1984). Social cognition. Reading, MA: Addison-Wesley.
16.
GoddardK.RoudsariA.WyattJ. C. (2012). Automation bias: A systematic review of frequency, effect mediators, and mitigators. Journal of the American Medical Informatics Association, 19(1), 121–127. doi:10.1136/amiajnl-2011-000089
17.
GopherD.BrauneR. (1984). On the psychophysics of workload: Why bother with subjective measures?Human Factors, 26, 519–532. doi:10.1177/001872088402600504
18.
HartS. G.StavelandL. E. (1988). Development of NASA-TLX (task load index): Results of empirical and theoretical research. Advances in Psychology, 52, 139–183.
19.
HayesA. F. (2006). A primer on multilevel modeling. Human Communication Research, 32(4), 385–410.
20.
HoffmanL.RovineM. J. (2007). Multilevel models for the experimental psychologist: Foundations and illustrative examples. Behavior Research Methods, 39(1), 101–117. doi:10.3758/bf03192848
21.
KimM. O.CoieraE.MagrabiF. (2017). Problems with health information technology and their effects on care delivery and patient outcomes: A systematic review. Journal of the American Medical Informatics Association, 24(2), 246–250. doi:10.1093/jamia/ocw154
22.
LeppinkJ.PaasF.van der VleutenC. P.van GogT.van MerriënboerJ. J. (2013). Development of an instrument for measuring different types of cognitive load. Behavior Research Methods, 45(4), 1058–1072.
23.
LeppinkJ.PaasF.van GogT.van der VleutenC. P.van MerriënboerJ. J. (2014). Effects of pairs of problems and examples on task performance and different types of cognitive load. Learning and Instruction, 30, 32–42.
24.
LyellD.CoieraE. (2017). Automation bias and verification complexity: A systematic review. Journal of the American Medical Informatics Association, 24(2), 423–431. doi:10.1093/jamia/ocw105
25.
LyellD.MagrabiF.RabanM. Z.PontL. G.BaysariM. T.DayR. O.CoieraE. (2017). Automation bias in electronic prescribing. BMC Medical Informatics and Decision Making, 17(1), 1–10. doi:10.1186/s12911-017-0425-5
26.
ManzeyD.ReichenbachJ.OnnaschL. (2012). Human performance consequences of automated decision aids: The impact of degree of automation and system experience. Journal of Cognitive Engineering and Decision Making, 6(1), 57–87. doi:10.1177/1555343411433844
27.
MetzgerU.ParasuramanR. (2005). Automation in future air traffic management: Effects of decision aid reliability on controller performance and mental workload. Human Factors, 47, 35–49. doi:10.1518/0018720053653802
28.
MillerG. A. (1956). The magical number seven, plus or minus two: Some limits on our capacity for processing information. Psychological Review, 63(2), 81–97.
29.
MolloyR.ParasuramanR. (1992, October12–16). Monitoring automation failures: Effects of automation reliability and task complexity. Paper presented at the Human Factors and Ergonomics Society annual meeting, Atlanta, GA.
30.
MosierK. L.SkitkaL. J. (1996). Human decision makers and automated decision aids: Made for each other. In ParasuramanR.MoulouaM. (Eds.), Automation and human performance: Theory and applications (pp. 201–220). Hillsdale, NJ: Lawrence Erlbaum.
31.
MosierK. L.SkitkaL. J.DunbarM.McDonnellL. (2001). Aircrews and automation bias: The advantages of teamwork?International Journal of Aviation Psychology, 11(1), 1–14. doi:10.1207/s15327108ijap1101_1
32.
MosierK. L.SkitkaL. J.HeersS.BurdickM. (1998). Automation bias: Decision making and performance in high-tech cockpits. International Journal of Aviation Psychology, 8(1), 47–63. doi:10.1207/s15327108ijap0801_3
33.
National Health and Medical Research Council, Australian Research Council, & Australian Vice-Chancellors’ Committee. (2015). National statement on ethical conduct in human research. Retrieved from www.nhmrc.gov.au/guidelines/publications/e72 (Original work published 2007)
34.
PaasF. (1992). Training strategies for attaining transfer of problem-solving skill in statistics: A cognitive-load approach. Journal of Educational Psychology, 84(4), 429–434. doi:10.1037/0022-0663.84.4.429
35.
PaasF.TuovinenJ. E.TabbersH.Van GervenP. W. M. (2003). Cognitive load measurement as a means to advance cognitive load theory. Educational Psychologist, 38(1), 63–71. doi:10.1207/S15326985EP3801_8
36.
ParasuramanR.ManzeyD. H. (2010). Complacency and bias in human use of automation: An attentional integration. Human Factors, 52, 381–410. doi:10.1177/0018720810376055
37.
ParasuramanR.MolloyR.SinghI. L. (1993). Performance consequences of automation-induced “complacency.”International Journal of Aviation Psychology, 3(1), 1–23. doi:10.1207/s15327108ijap0301_1
38.
PetersonL.PetersonM. J. (1959). Short-term retention of individual verbal items. Journal of Experimental Psychology, 58(3), 193–198. doi:10.1037/h0049234
39.
PeughJ. L. (2010). A practical guide to multilevel modeling. Journal of School Psychology, 48(1), 85–112. doi:10.1016/j.jsp.2009.09.002
40.
PovyakaloA. A.AlberdiE.StriginiL.AytonP. (2013). How to discriminate between computer-aided and computer-hindered decisions: A case study in mammography. Medical Decision Making, 33(1), 98–107. doi:10.1177/0272989x12465490
41.
PrinzelL. J.IIIFreemanF. G.PrinzelH. D. (2005). Individual differences in complacency and monitoring for automation failures. Individual Differences Research, 3(1), 27–49.
42.
ReichenbachJ.OnnaschL.ManzeyD. (2011). Human performance consequences of automated decision aids in states of sleep loss. Human Factors, 53, 717–728. doi:10.1177/0018720811418222
43.
RichardsonJ. T. E. (2011). Eta squared and partial eta squared as measures of effect size in educational research. Educational Research Review, 6(2), 135–147. doi:10.1016/j.edurev.2010.12.001
44.
RoviraE.McGarryK.ParasuramanR. (2007). Effects of imperfect automation on decision making in a simulated command and control task. Human Factors, 49, 76–87. doi:10.1518/001872007779598082
45.
SinghA. L.TiwariT.SinghI. L. (2009). Effects of automation reliability and training on automation-induced complacency and perceived mental workload. Journal of the Indian Academy of Applied Psychology, 35, 9–22.
46.
SinghI. L.SharmaH. O.ParasuramanR. (2001). Effects of manual training and automation reliability on automation induced complacency in flight simulation task. Psychological Studies, 46(1/2), 21–27.
47.
SinghI. L.SinghA. L.SahaP. K. (2007, July22–27). Monitoring performance and mental workload in an automated system. Paper presented at the International Conference on Engineering Psychology and Cognitive Ergonomics, Beijing, China.
48.
SintchenkoV.CoieraE. W. (2003). Which clinical decisions benefit from automation? A task complexity approach. International Journal of Medical Informatics, 70(2/3), 309–316. doi:10.1016/S1386-5056(03)00040-6
49.
SkitkaL. J.MosierK.BurdickM. D. (2000). Accountability and automation bias. International Journal of Human Computer Studies, 52(4), 701–717. doi:10.1006/ijhc.1999.0349
50.
SweidanM.WilliamsonM.ReeveJ. F.HarveyK.O’NeillJ. A.SchattnerP.SnowdonT. (2011). Evaluation of features to support safety and quality in general practice clinical software. BMC Medical Informatics and Decision Making, 11(1), 1–8. doi:10.1186/1472-6947-11-27
51.
SwellerJ. (1994). Cognitive load theory, learning difficulty, and instructional design. Learning and Instruction, 4(4), 295–312. doi:10.1016/0959-4752(94)90003-5
52.
SwellerJ.AyresP.KalyugaS. (2011). Cognitive load theory (Vol. 1). New York, NY: Springer.
53.
TwiskJ. (2006). Applied multilevel analysis: A practical guide. Cambridge, UK: Cambridge University Press.
54.
van RosseF.MaatB.RademakerC. M. A.van VughtA. J.EgbertsA. C. G.BollenC. W. (2009). The effect of computerized physician order entry on medication prescription errors and clinical outcome in pediatric and intensive care: A systematic review. Pediatrics, 123(4), 1184–1190. doi:10.1542/peds.2008-1494
55.
WickensC. D.CleggB. A.VieaneA. Z.SebokA. L. (2015). Complacency and automation bias in the use of imperfect automation. Human Factors, 57, 728–739. doi:10.1177/0018720815581940
56.
WolfstadtJ. I.GurwitzJ. H.FieldT. S.LeeM.KalkarS.WuW.RochonP. A. (2008). The effect of computerized physician order entry with clinical decision support on the rates of adverse drug events: A systematic review. Journal of General Internal Medicine, 23(4), 451–458. doi:10.1007/s11606-008-0504-5
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
