The objective of this study was to test the predictions of the routine-failure trade-off (or lumberjack) model in a full-scope simulator study with expert operators performing realistic control tasks.
Background
A meta-study of degree of automation (DOA) studies concluded that DOA predicts task performance under both routine and automation failure conditions, workload, and situation awareness. Empirical support for this conclusion appears to be weak for complex work situations.
Method
A full-scope nuclear power plant simulator experiment was conducted in which licensed operating crews completed realistic procedure execution tasks. Dependent measures selected from the lumberjack model were collected and analyzed for systematic effects.
Results
Situation awareness increased with increasing DOA, which contradicts the lumberjack model. Anticipated workload and failure task performance effects were not observed.
Conclusion
The experimental results add further evidence challenging the applicability of the lumberjack model to complex work situations.
Application
Practitioners should use caution when extending the predictions of the lumberjack model based on data from simple work situations to complex work situations. Researchers should invest more resources in testing the predictive power of the lumberjack model in complex work situations.
AndresenG.SvengrenH.HeimdalJ. O.NilsenS.HulsundJ. E.BisioR.DebroiseX. (2004). Procedure automation: The effect of automated procedure execution on situation awareness and human performance (HWR-759). Halden, Norway: OECD Halden Reactor Project.
2.
BisioR.HulsundJ. E.NilsenS. (2001). COPMA-III: A tool for developing effective computerised procedure systems (HWR-644). Halden, Norway: OECD Halden Reactor Project.
3.
BraarudP. Ø. (1998). Complexity factors and predictions of crew performance (HWR-521). Halden, Norway: OECD Halden Reactor Project.
4.
BraarudP. Ø. (2000). Subjective task complexity in the control room (HWR-621). Halden, Norway: OECD Halden Reactor Project.
BraarudP. Ø.LudvigsenJ. T. (2002). Experimental study of the effect of task priority and coordination strategy on crew performance (HWR-704). Halden, Norway: OECD Halden Reactor Project.
7.
BrandenburgerN.JippM. (2017). Effects of expertise for automatic train operations. Cognition, Technology & Work, 19, 699–709.
8.
BrasethA. O.NihlwingC.SvengrenH.VelandØ.HurlenL.KvalemJ. (2009). Lessons learned from Halden project research on human system interfaces. Nuclear Engineering and Technology, 41, 215–224.
9.
BurnsC. M.SkraaningG.Jr.JamiesonG. A.LauN.KwokJ.WelchR.AndresenG. (2008). Evaluation of ecological interface design for nuclear process control: Situation awareness effects. Human Factors, 50, 663–679.
10.
CalhounG. L.DraperM. K.RuffH. A. (2009). Effect of level of automation on unmanned aerial vehicle routing task. In Proceedings of the Human Factors and Ergonomics Society 53rd Annual Meeting (pp. 197–201). Santa Monica, CA: Human Factors and Ergonomics Society.
11.
CopiI. (1953). Introduction to logic. New York, NY: Macmillan.
12.
CummingsM. L.MitchellP. J. (2007). Operator scheduling strategies in supervisory control of multiple UAVs. Aerospace Science and Technology, 11, 339–348.
13.
EndsleyM. R. (2016). Designing for situation awareness: An approach to user-centered design. Boca Raton, FL: CRC Press.
14.
EndsleyM. R. (2018). Level of automation forms a key aspect of autonomy design. Journal of Cognitive Engineering and Decision Making, 12, 29–34.
15.
EndsleyM. R.KaberD. (1999). Level of automation effects on performance, situation awareness and workload in a dynamic control task. Ergonomics, 42, 462–492.
16.
FernandesA.BraarudP. Ø. (2015). Exploring measures of workload, situation awareness, and task performance in the main control room. Procedia Manufacturing, 3, 1281–1288.
17.
ForesterJ.DangV. N.ByeA.LoisE.MassaiuS.BrobergH.. . . JuliusJ. (2014). The international HRA empirical study: Lessons learned from comparing HRA methods predictions to HAMMLAB simulator data (NUREG-2127). Washington, DC: U.S. Nuclear Regulatory Commission.
18.
HolmstroemC. O.GreenM. (2000). Human factors research at the OECD Halden Reactor Project into the 21st century. In Proceedings of the Human Factors and Ergonomics Society Annual Meeting (Vol. 44, No. 22, pp. 803–806). Los Angeles, CA: SAGE.
19.
JamiesonG. A.SkraaningG. (2018). Levels of automation in human factors models for automation design: Why we might consider throwing the baby out with the bathwater. Journal of Cognitive Engineering and Decision Making, 12, 42–49.
20.
KaberD. B. (2018). Reflections on commentaries on “issues in human–automation interaction modeling.”Journal of Cognitive Engineering and Decision Making, 12, 86–93.
21.
KaberD. B.EndsleyM. R. (2004). The effects of level of automation and adaptive automation on human performance, situation awareness and workload in a dynamic control task. Theoretical Issues in Ergonomics Science, 5, 113–153.
22.
LauN.JamiesonG. A.SkraaningG.Jr. (2014). Inter-rater reliability of query/probe-based techniques for measuring situation awareness. Ergonomics, 57, 959–972.
23.
LauN.JamiesonG. A.SkraaningG. (2016). Empirical evaluation of the Process Overview Measure for assessing situation awareness in process plants. Ergonomics, 59, 393–408.
24.
LauN.JamiesonG. A.SkraaningG.Jr.BurnsC. M. (2008). Ecological interface design in the nuclear domain: An empirical evaluation of ecological displays for the secondary subsystems of a boiling water reactor plant simulator. IEEE Transactions on Nuclear Science, 55, 3597–3610.
25.
LauN.SkraaningG.Jr.JamiesonG. A. (2009, August). Metacognition in nuclear process control. In Proceedings of the 17th Triennial World Congress on Ergonomics. Beijing, China: IEA.
26.
MetzgerU.ParasuramanR. (2005). Automation in future air traffic management: Effects of decision aid reliability on controller performance and mental workload. Human Factors, 47, 35–49.
27.
OckermanJ.PritchettA. (2000). A review and reappraisal of task guidance: Aiding workers in procedure following. International Journal of Cognitive Ergonomics, 4, 191–212.
28.
O’HaraJ. M.BrownW. S.HallbertB.SkraaningG.PersenskyJ. J.WachtelJ. (2000). The effects of alarm display, processing, and availability on crew performance (NUREG/CR-6691). Washington, DC: U.S. Nuclear Regulatory Commission.
29.
O’HaraJ. M.HigginsJ. C.StublerW. F.KramerJ. (2000). Computer-based procedure systems: Technical basis and human factors review guidance (NUREG/CR-6634). Washington, DC: U.S. Nuclear Regulatory Commission.
30.
OnnaschL.WickensC. D.LiH.ManzeyD. (2014). Human performance consequences of stages and levels of automation: An integrated meta-analysis. Human Factors, 56, 476–488.
31.
OwreF.KvalemJ.KarlssonT.NihlwingC. (2002). A new integrated BWR supervision and control system. In Proceedings of the 2002 IEEE 7th Conference on Human Factors and Power Plants (pp. 4-41–4-47). Piscataway, NJ: IEEE.
32.
ParasuramanR.SheridanT. B.WickensC. D. (2000). A model for types and levels of human interaction with automation. IEEE Transactions on Systems, Man, and Cybernetics-Part A: Systems and Humans, 30(3), 286–297.
33.
PopperK. R. (2005). The logic of scientific discovery. London, England: Taylor & Francis. (Original work published 1935)
34.
SarterN. B.SchroederB. (2001). Supporting decision making and action selection under time pressure and uncertainty: The case of in-flight icing. Human Factors, 43, 573–583.
35.
SebokA.WickensC. D. (2017). Implementing lumberjacks and black swans into model-based tools to support human-automation interaction. Human Factors, 59, 189–203.
36.
SkjerveA. B.ByeA. (Eds.). (2011). Simulator-based human factors studies across 25 years. London, England: Springer.
37.
SkjerveA.NihlwingC.NystadE.StrandS. (2009). Lessons learned from the extended teamwork study on work organization, collaboration technologies, and automatic agents. In Proceedings of 17th World Congress on Ergonomics. Beijing, China: IEA.
38.
SkraaningG. (1998). The Operator Performance Assessment System (OPAS) (HWR-538). Halden, Norway: OECD Halden Reactor Project.
39.
SkraaningG. (2003). Experimental control versus realism: Methodological solutions for simulator studies in complex operating environments (OECD Halden Reactor Project. HPR-361, Dr.philos. dissertation). Norwegian University of Science and Technology, Trondheim.
40.
SkraaningG.ByeA. (2011). The purpose of HAMMLAB and the theoretical basis for experimental research. In SkjerveA. B.ByeA. (Eds.), Simulator-based human factors studies across 25 years (pp. 45–56). London, England: Springer.
41.
SkraaningG.EitrheimM. H. R. (2011). Methodological challenges in HAMMLAB. In SkjerveA. B.ByeA. (Eds.), Simulator-based human factors studies across 25 years (pp. 57–74). London, England: Springer.
42.
SkraaningG.EitrheimM. H. R.LauN.NihlwingC.HurlenL.KarlssonT. (2010). Coping with automation in future plants: Results from the 2009 HAMMLAB experiment (HWR-937). Halden, Norway: OECD Halden Reactor Project.
43.
SkraaningG.JamiesonG. A. (2017). Twenty years of HRP research on human-automation interaction: Insights on automation transparency and levels of automation (HPR-387). Halden, Norway: OECD Halden Reactor Project.
SkraaningG.SkjerveA. B. (2006, November). Trust in automation and metacognitive accuracy in NPP operating crews. Proceedings of the Fifth American Nuclear Society International Topical Meeting on Nuclear Plant Instrumentation, Controls and Human-Machine Interface Technologies (NPIC&HMIT 2006), Albuquerque, NM.
46.
StublerW. F.O’HaraJ. M.KramerJ. (2000). Soft controls: Technical basis and human factors review guidance (No. NUREG/CR-6635). Upton. NY: Brookhaven National Laboratory.
47.
VicenteK. J. (1999). Cognitive work analysis: Toward safe, productive, and healthy computer-based work. Boca Raton, FL: CRC Press.
48.
WickensC. D. (2018). Automation stages & levels, 20 years after. Journal of Cognitive Engineering and Decision Making, 12, 35–41.
49.
WickensC. D.LiH.SantamariaA.SebokA.SarterN. B. (2010). Stages and levels of automation: An integrated meta-analysis. In Proceedings of the Human Factors and Ergonomics Society Annual Meeting (Vol. 54, No. 4, pp. 389–393). Los Angeles, CA: SAGE.
50.
WrightM. C.KaberD. B.EndsleyM. R. (2003). Performance and situation awareness effects of levels of automation in an advanced commercial aircraft flight simulation. In Proceedings of the 12th International Symposium on Aviation Psychology (pp. 1277–1282). Dayton, OH: Wright State University.
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
