Where Is the Function Allocation Boundary? The Effect of Degree of Automation on Attention Allocation and Human Performance Under Different Reliabilities

Abstract

Objective

This study investigated the effect of reliability on the function allocation (FA) boundary by examining the interaction effect of degree of automation (DOA) and reliability on routine performance, failure performance, and attention allocation.

Background

According to the lumberjack effect, an increase in DOA will typically improve routine performance, while failure performance may remain undeteriorated until a specific, high DOA threshold is reached. This threshold can be regarded as the FA boundary. Considering that both DOA and reliability can influence failure performance through attention allocation, it is crucial to investigate how reliability affects the FA boundary.

Method

Participants performed three MATB tasks, one of which, the system monitoring task, was supported by four types of automation: information acquisition (IAc), information analysis (IAn), action selection (AS), and action implementation (AI). From IAc to AI, the DOA incrementally increased. Additionally, automation reliability was set to three levels, namely, 87.50%, 68.75%, and 56.25%.

Results

For routine performance, participants assisted by AS reacted more rapidly to gauge malfunctions than those supported by IAc or IAn. For failure performance, participants aided by AI corrected gauge malfunctions less frequently than other participants. Correspondingly, participants supported by AI exhibited fewer fixation counts on the system monitoring task than did others.

Conclusion

It appears that the FA boundary lies between AS and AI. However, there is insufficient evidence to support the effect of reliability on the FA boundary.

Application

These findings can provide useful insights for improving the design of automated systems in complex working environments.

Keywords

automation support routine performance failure performance fixation count

Get full access to this article

View all access options for this article.

References

Altmann

E. M.

(2004). The preparation effect in task switching: Carryover of SOA. Memory & Cognition, 32(1), 153–163. https://doi.org/10.3758/BF03195828

Avril

(2023). Providing different levels of accuracy about the reliability of automation to a human operator: Impact on human performance. Ergonomics, 66(2), 217–226. https://doi.org/10.1080/00140139.2022.2069870

Avril

Cegarra

Wioland

Navarro

(2022). Automation type and reliability impact on visual automation monitoring and human performance. International Journal of Human-Computer Interaction, 38(1), 64–77. https://doi.org/10.1080/10447318.2021.1925435

Avril

Valéry

Navarro

Wioland

Cegarra

(2021). Effect of imperfect information and action automation on attentional allocation. International Journal of Human-Computer Interaction, 37(11), 1063–1073. https://doi.org/10.1080/10447318.2020.1870817

Bagheri

Jamieson

G. A.

(2004). Considering subjective trust and monitoring behavior in assessing automation-induced “complacency”. Human performance, situation awareness, and automation: Current research and trends, 1, 54–59.

Cegarra

Baracat

Calmettes

Matton

Capa

R. L.

(2017). A neuroergonomics perspective on mental workload predictions in Jens Rasmussen’s SRK framework. Travail Humain, Le, 80(1), 7–22. https://doi.org/10.3917/th.801.0007

Cegarra

Valéry

Avril

Calmettes

Navarro

(2020). OpenMATB: A multi-attribute task battery promoting task customization, software extensibility and experiment replicability. Behavior Research Methods, 52(5), 1980–1990. https://doi.org/10.3758/s13428-020-01364-w

Charness

Gneezy

Kuhn

M. A.

(2012). Experimental methods: Between-subject and within-subject design. Journal of Economic Behavior & Organization, 81(1), 1–8. https://doi.org/10.1016/j.jebo.2011.08.009

Deroo

Hoc

J. M.

Mars

(2013). Effect of strength and direction of haptic cueing on steering control during near lane departure. Transportation Research Part F: Traffic Psychology and Behaviour, 16(1), 92–103. https://doi.org/10.1016/j.trf.2012.08.015

10.

Endsley

M. R.

(2017). From here to autonomy: Lessons learned from human–automation research. Human Factors, 59(1), 5–27. https://doi.org/10.1177/0018720816681350

11.

Faul

Erdfelder

Lang

A. G.

Buchner

(2007). G* Power 3: A flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behavior Research Methods, 39(2), 175–191. https://doi.org/10.3758/bf03193146

12.

Feigh

K. M.

Dorneich

M. C.

Hayes

C. C.

(2012). Toward a characterization of adaptive systems: A framework for researchers and system designers. Human Factors, 54(6), 1008–1024. https://doi.org/10.1177/0018720812443983

13.

Ferraro

Clark

Christy

Mouloua

(2018). Effects of automation reliability and trust on system monitoring performance in simulated flight tasks. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 62(1), 1232–1236. https://doi.org/10.1177/1541931218621283

14.

Ferraro

J. C.

Mouloua

(2021). Effects of automation reliability on error detection and attention to auditory stimuli in a multi-tasking environment. Applied Ergonomics, 91(6), Article 103303. https://doi.org/10.1016/j.apergo.2020.103303

15.

Foroughi

C. K.

Devlin

Pak

Brown

N. L.

Sibley

Coyne

J. T.

(2023). Near-perfect automation: Investigating performance, trust, and visual attention allocation. Human Factors, 65(4), 546–561. https://doi.org/10.1177/00187208211032889

16.

Galster

S. M.

(2003). An examination of complex human-machine system performance under multiple levels and stages of automation. The Catholic University of America.

17.

Haslbeck

Hoermann

H. J.

(2016). Flying the needles: Flight deck automation erodes fine-motor flying skills among airline pilots. Human Factors, 58(4), 533–545. https://doi.org/10.1177/0018720816640394

18.

Hasse

Bruder

(2015). Eye-tracking measurements and their link to a normative model of monitoring behaviour. Ergonomics, 58(3), 355–367. https://doi.org/10.1080/00140139.2014.967310

19.

Hoesterey

Onnasch

(2023). The effect of risk on trust attitude and trust behavior in interaction with information and decision automation. Cognition, Technology & Work, 25(1), 15–29. https://doi.org/10.1007/s10111-022-00718-y

20.

Huegli

Merks

Schwaninger

(2020). Automation reliability, human–machine system performance, and operator compliance: A study with airport security screeners supported by automated explosives detection systems for cabin baggage screening. Applied Ergonomics, 86, Article 103094. https://doi.org/10.1016/j.apergo.2020.103094

21.

Karpinsky

N. D.

Chancey

E. T.

Palmer

D. B.

Yamani

(2018). Automation trust and attention allocation in multitasking workspace. Applied Ergonomics, 70, 194–201. https://doi.org/10.1016/j.apergo.2018.03.008

22.

Wickens

C. D.

Sarter

Sebok

(2014). Stages and levels of automation in support of space teleoperations. Human Factors, 56(6), 1050–1061. https://doi.org/10.1177/0018720814522830

23.

W. C.

Zhang

Court

Kearney

Braithwaite

(2022). The influence of augmented reality interaction design on Pilot’s perceived workload and situation awareness. International Journal of Industrial Ergonomics, 92(3), Article 103382. https://doi.org/10.1016/j.ergon.2022.103382

24.

Zhang

Feldhütter

Happee

Martens

De Winter

J. C.

(2019). Beyond mere take-over requests: The effects of monitoring requests on driver attention, take-over performance, and acceptance. Transportation Research Part F: Traffic Psychology and Behaviour, 63, 22–37. https://doi.org/10.1016/j.trf.2019.03.018

25.

Zhang

Yang

Kang

Chai

Shi

Zeng

(2021). Take over gradually in conditional automated driving: The effect of two-stage warning systems on situation awareness, driving stress, takeover performance, and acceptance. International Journal of Human-Computer Interaction, 37(4), 352–362. https://doi.org/10.1080/10447318.2020.1860514

26.

Manzey

Reichenbach

Onnasch

(2012). Human performance consequences of automated decision aids: The impact of degree of automation and system experience. Journal of Cognitive Engineering and Decision Making, 6(1), 57–87. https://doi.org/10.1177/1555343411433844

27.

Miller

C. A.

Parasuraman

(2007). Designing for flexible interaction between humans and automation: Delegation interfaces for supervisory control. Human Factors, 49(1), 57–75. https://doi.org/10.1518/001872007779598037

28.

Oakley

Mouloua

Hancock

(2003). Effects of automation reliability on human monitoring performance. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 47(1), 188–190. https://doi.org/10.1177/154193120304700139

29.

Onnasch

(2015). Crossing the boundaries of automation—function allocation and reliability. International Journal of Human-Computer Studies, 76, 12–21. https://doi.org/10.1016/j.ijhcs.2014.12.004

30.

Onnasch

Ruff

Manzey

(2014). Operators׳ adaptation to imperfect automation–Impact of miss-prone alarm systems on attention allocation and performance. International Journal of Human-Computer Studies, 72(10-11), 772–782. https://doi.org/10.1016/j.ijhcs.2014.05.001

31.

Onnasch

Wickens

C. D.

Manzey

(2014). Human performance consequences of stages and levels of automation: An integrated meta-analysis. Human Factors, 56(3), 476–488. https://doi.org/10.1177/0018720813501549

32.

Palada

Neal

Tay

Heathcote

(2018). Understanding the causes of adapting, and failing to adapt, to time pressure in a complex multistimulus environment. Journal of Experimental Psychology: Applied, 24(3), 380–399. https://doi.org/10.1037/xap0000176

33.

Parasuraman

Manzey

D. H.

(2010). Complacency and bias in human use of automation: An attentional integration. Human Factors, 52(3), 381–410. https://doi.org/10.1177/0018720810376055

34.

Parasuraman

Molloy

Singh

I. L.

(1993). Performance consequences of automation-induced ‘complacency. The International Journal of Aviation Psychology, 3(1), 1–23. https://doi.org/10.1207/s15327108ijap0301_1

35.

Parasuraman

Sheridan

T. B.

Wickens

C. D.

(2000). A model for types and levels of human interaction with automation. IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans: A Publication of the IEEE Systems, Man, and Cybernetics Society, 30(3), 286–297. https://doi.org/10.1109/3468.844354

36.

Rasmussen

Vicente

K. J.

(1989). Coping with human errors through system design: Implications for ecological interface design. International Journal of Man-Machine Studies, 31(5), 517–534. https://doi.org/10.1016/0020-7373(89)90014-X

37.

Rieth

Hagemann

(2022). Automation as an equal team player for humans? A view into the field and implications for research and practice. Applied Ergonomics, 98(4), Article 103552. https://doi.org/10.1016/j.apergo.2021.103552

38.

Rovira

McGarry

Parasuraman

(2007). Effects of imperfect automation on decision making in a simulated command and control task. Human Factors, 49(1), 76–87. https://doi.org/10.1518/001872007779598082

39.

Rovira

Pak

McLaughlin

(2017). Effects of individual differences in working memory on performance and trust with various degrees of automation. Theoretical Issues in Ergonomics Science, 18(6), 573–591. https://doi.org/10.1080/1463922X.2016.1252806

40.

Rovira

Zinni

Parasuraman

(2002). Effects of information and decision automation on multi-task performance. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 46(3), 327–331. https://doi.org/10.1177/154193120204600324

41.

Sarter

N. B.

Schroeder

(2001). Supporting decision making and action selection under time pressure and uncertainty: The case of in-flight icing. Human Factors, 43(4), 573–583. https://doi.org/10.1518/001872001775870403

42.

Sebok

Wickens

C. D.

(2017). Implementing lumberjacks and black swans into model-based tools to support human–automation interaction. Human Factors, 59(2), 189–203. https://doi.org/10.1177/0018720816665201

43.

Smith

E. E.

(1968). Choice reaction time: An analysis of the major theoretical positions. Psychological Bulletin, 69(2), 77–110. https://doi.org/10.1037/h0020189

44.

Tatasciore

Bowden

V. K.

Visser

T. A.

Loft

(2022). Should we just let the machines do it? The benefit and cost of action recommendation and action implementation automation. Human Factors, 64(7), 1121–1136. https://doi.org/10.1177/0018720821989148

45.

Tatasciore

Bowden

V. K.

Visser

T. A.

Michailovs

S. I.

Loft

(2020). The benefits and costs of low and high degree of automation. Human Factors, 62(6), 874–896. https://doi.org/10.1177/0018720819867181

46.

Taylor

R. M.

(2017). Situational awareness rating technique (SART): The development of a tool for aircrew systems design. In Situational awareness (pp. 111–128). Routledge.

47.

Volz

K. M.

Dorneich

M. C.

(2020). Evaluation of cognitive skill degradation in flight planning. Journal of Cognitive Engineering and Decision Making, 14(4), 263–287. https://doi.org/10.1177/1555343420962897

48.

Wang

Yang

Liu

Cao

(2014). An eye-tracking study of website complexity from cognitive load perspective. Decision Support Systems, 62(2), 1–10. https://doi.org/10.1016/j.dss.2014.02.007

49.

Wang

Liu

You

(2023). The effect of two-stage warning system on human performance along with different takeover strategies. International Journal of Industrial Ergonomics, 97(2), Article 103492. https://doi.org/10.1016/j.ergon.2023.103492

50.

Wickens

(2018). Automation stages & levels, 20 years after. Journal of Cognitive Engineering and Decision Making, 12(1), 35–41. https://doi.org/10.1177/1555343417727438

51.

Wickens

C. D.

Clegg

B. A.

Vieane

A. Z.

Sebok

A. L.

(2015). Complacency and automation bias in the use of imperfect automation. Human Factors, 57(5), 728–739. https://doi.org/10.1177/0018720815581940

52.

Wickens

C. D.

Dixon

S. R.

(2007). The benefits of imperfect diagnostic automation: A synthesis of the literature. Theoretical Issues in Ergonomics Science, 8(3), 201–212. https://doi.org/10.1080/14639220500370105

53.

Wickens

C. D.

Santamaria

Sebok

Sarter

N. B.

(2010). Stages and levels of automation: An integrated meta-analysis. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 54(4), 389–393. https://doi.org/10.1177/154193121005400425

54.

Wiczorek

Manzey

(2014). Supporting attention allocation in multitask environments: Effects of likelihood alarm systems on trust, behavior, and performance. Human Factors, 56(7), 1209–1221. https://doi.org/10.1177/0018720814528534

55.

Yuan

(2023). Overcoming underload and overload: The effect of stage and level of autonomy in routine-failure trade-off in human-autonomy teams. International Journal of Industrial Ergonomics, 94(9), Article 103424. https://doi.org/10.1016/j.ergon.2023.103424

56.

Yamani

Horrey

W. J.

(2018). A theoretical model of human-automation interaction grounded in resource allocation policy during automated driving. International Journal of Human Factors and Ergonomics, 5(3), 225–239. https://doi.org/10.1504/IJHFE.2018.095912

57.

Zhou

Cui

(2022). The effect of automation trust tendency, system reliability and feedback on users’ phishing detection. Applied Ergonomics, 102(11), Article 103754. https://doi.org/10.1016/j.apergo.2022.103754

58.

Zirk

Wiczorek

Manzey

(2020). Do we really need more stages? Comparing the effects of likelihood alarm systems and binary alarm systems. Human Factors, 62(4), 540–552. https://doi.org/10.1177/0018720819852023