Recent successes in artificial intelligence (AI) have ignited debate over its role in human-machine systems—specifically, whether an AI system should be viewed as a tool or a teammate. This article consists of a set of essays that explore this question from a macrocognitive viewpoint. These essays reveal similarities across stances, and divergences regarding interpretations of the teammate metaphor. Discussions include concerns about potential risks and effects of the teammate metaphor on users, in addition to expressions of the value of the metaphor for work system design. The essays highlight the role of metaphors in both enhancing and stifling design creativity, the blending of metaphors in design, and the need for empirical evaluation to guide clear design choices. Moreover, regardless of which approach is adopted, the “technology-first” mindset of system developers, the emphasis on relatively simple human-machine interactions over complex, evolving work systems, and the insidious effects of AI on human expertise all challenge progress in the design of macrocognitive work systems. The essays present some specific research directions.
BansalG.WuT.ZhouJ.FokR.NushiB.KamarE.RibeiroM. T.WeldD. (2021). Does the whole exceed its parts? The effect of AI explanations on complementary team performance. In CHI '21: Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (pp, 1–16). ACM. https://doi.org/10.1145/3411764.3445717
3.
BarleyS. R. (1990). The alignment of technology and structure through roles and networks. Administrative Science Quarterly, 35(1), 61–103.
4.
BartlettM. L.McCarleyJ. S. (2024). Experts and non-experts use diagnostic aids inefficiently. Proceedings of the Human Factors and Ergonomics Society - Annual Meeting, 68(1), 98–103. https://doi.org/10.1177/10711813241275516
5.
BechkyB. A.OkhuysenG. A. (2011). Expecting the unexpected? How SWAT officers and film crews handle surprises. Academy of Management Journal, 54(2), 239–261. https://doi.org/10.5465/amj.2011.60263060
6.
BigleyG. A.RobertsK. H. (2001). The incident command system: High-reliability organizing for complex and volatile task environments. Academy of Management Journal, 44(6), 1281–1299. https://doi.org/10.2307/3069401
7.
BogdanovicJ.PerryJ.GuggenheimM.ManserT. (2015). Adaptive coordination in surgical teams. BMC Health Services Research, 15(1), 128. https://doi.org/10.1186/s12913-015-0792-5
8.
BroussardM. (2018). Artificial Unintelligence: How computers misunderstand the world. MIT Press.
9.
BuçincaZ.MalayaM. B.I GajosK. Z. (2021). To trust or to think: Cognitive forcing functions can reduce overreliance on AI in AI-assisted decision-making. Proc. ACM Hum.-Comput. Interact, 5(CSCW1):1–21. https://doi.org/10.1145/3449287
10.
CalhounG. (2022). Adaptable (not adaptive) automation: Forefront of human-automation teaming. Human Factors, 64(2), 269–277. https://doi.org/10.1177/00187208211037457
11.
CarrollJ. M. (1988). Mental models in human-computer interaction. In Handbook of human-computer interaction (pp. 45–65). Elsevier.
12.
CastellinoR. A. (2005). Computer aided detection (CAD): An overview. Cancer Imaging: The Official Publication of the International Cancer Imaging Society, 5(1), 17–19. https://doi.org/10.1102/1470-7330.2005.0018
13.
ChristoffersenK.WoodsD. (2001). How to make automated systems team players. Advances in Human Performance and Cognitive Engineering Research, 2, 1–12. https://doi.org/10.1016/S1479-3601(02)02003-9
14.
ClanceyW. J. (1987). Knowledge-based tutoring. MIT Press.
15.
CochranA.RayoM. F. (2023). Toward joint activity design: Augmenting user-centered design with Heuristics for supporting joint activity. Proceedings of the International Symposium of Human Factors and Ergonomics in Healthcare. International Symposium of Human Factors and Ergonomics in Healthcare, 12(1), 19–23. https://doi.org/10.1177/2327857923121006
16.
CookeN. J.CohenM. C.FazioW. C.InderbergL. H.JohnsonC. J.LemattaG. J.PeelM.TeoA. (2024). From teams to teamness: Future directions in the science of team cognition. Human Factors, 66(6), 1669–1680. https://doi.org/10.1177/00187208231162449
17.
CooleyM. (1989). Human-centered systems: Designing human-centered technology (pp. 133–143). The Springer Series on Artificial Intelligence and Society.
DominguezC.McDermottP.WardP. (2022). Beyond the loop: Redefining models of human-autonomy interaction. International Conference on Naturalistic Decision Making.
EkelhofM. A. C. (2018). Lifting the fog of targeting: “Autonomous weapons” and human control through the lens of military targeting. Naval War College Review, 71(3), 6.
24.
ElixB.NaikarN. (2021). Designing for adaptation in workers' individual behaviors and collective structures with cognitive work analysis: Case study of the diagram of work organization possibilities. Human Factors, 63(2), 274–295. https://doi.org/10.1177/0018720819893510
25.
EndsleyM. (2023). Supporting human-AI teams: Transparency, explainability, and situation awareness. Computers in Human Behavior, 140(3), Article 107574. https://doi.org/10.1016/j.chb.2022.107574
26.
EndsleyM. R. (2017). Autonomous driving systems: A preliminary naturalistic study of the tesla model S. Journal of Cognitive Engineering and Decision Making, 11(3), 225–238. https://doi.org/10.1177/1555343417695197
27.
EskridgeT. C.StillD.HoffmanR. R. (2014). Principles for human-centered interaction design, Part 1: Performative Systems. IEEE Intelligent Systems, 29(04), 88–94. https://doi.org/10.1109/mis.2014.68
28.
FittsP. M. (Ed.), (1951). Human engineering for an effective air navigation and traffic control system. National Research Council.
29.
ForbusK. D.FeltovichP. J. (Eds.), (2001). Smart machines in education. AAAI Press.
30.
GroomV.NassC. (2007). Can robots be teammates?Interaction Studies. Social Behaviour and Communication in Biological and Artificial Systems, 8(3), 483–500. https://doi.org/10.1075/is.8.3.10gro
HeltonW. S. (2009). Canine ergonomics: The science of working dogs. Taylor & Francis.
33.
HoffK. A.BashirM. (2015). Trust in automation: Integrating empirical evidence on factors that influence trust. Human Factors, 57(3), 407–434. https://doi.org/10.1177/0018720814547570
34.
HoffmanR.BannonL. J.SebeN. (2011). Human-centered computing. In Encyclopedia of software engineering (pp. 395–403). Taylor & Francis.
35.
HoffmanR. R. (1980). Metaphor in science. In Cognition and figurative language (pp. 393–423). Erlbaum.
HutchinsE. (1995). Cognition in the wild. MIT Press.
38.
JacobsM.PradierM. F.McCoyT. H.PerlisR. H.Doshi-VelezF.GajosK. Z. (2021). How machine-learning recommendations influence clinician treatment selections: The example of antidepressant selection. Translational Psychiatry, 11(1), 108. https://doi.org/10.1038/s41398-021-01224-xe
39.
JeffriesR. (1997). Position paper. In FlanaganJ.HuangT.JonesP.KasifS. (Eds.), Human-centered systems: Information, interactivity and intelligence (pp. 277–279). National Science Foundation.
40.
JohnsonM.BradshawJ. (2021). How interdependence explains the world of teamwork. In LawlessW. F.LinasJ.SofgeD. A. (Eds.), Engineering artificially intelligent systems (pp. 122–146). Springer.
41.
JohnsonM.BradshawJ.FeltovichP.JonkerC.vanR.SierhuisM. (2014). Coactive design: Designing support for interdependence in joint activity. Journal of Human-Robot Interaction, 3(1), 43–69. https://doi.org/10.5898/JHRI.3.1.Johnson
KapaloK. A.PhillipsE.FioreS. M. (2016). The application and extension of the human-animal team model to better understand human-robot interaction: Recommendations for further research. Proceedings of the Human Factors and Ergonomics Society - Annual Meeting, 60(1), 1225–1229. https://doi.org/10.1177/1541931213601286
44.
KirlikA. (1993). Modeling strategic behavior in human-automation interaction: Why an “aid” can and should go unused. Human Factors, 35(2), 221–242. https://doi.org/10.1177/001872089303500203
45.
KleinG. (2005). The power of intuition. A Currency Book/Doubleday.
KleinG.WoodsD. D.BradshawJ. D.HoffmanR. R.FeltovichP. J. (2004). Ten challenges for making automation a “team player” in joint human-agent activity. IEEE Intelligent Systems, 6(06), 91–95. https://doi.org/10.1109/mis.2004.74
LeeJ.RheemH.LeeJ. D.SzczerbaJ. F.TsimhoniO. (2023). Teaming with your car: Redefining the driver–automation relationship in highly automated vehicles. Journal of Cognitive Engineering and Decision Making, 17(1), 49–74. https://doi.org/10.1177/15553434221132636
50.
LundbergJ.RankinA. (2014). Resilience and vulnerability of small flexible crisis response teams: Implications for training and preparation. Cognition, Technology & Work, 16(2), 143–155. https://doi.org/10.1007/s10111-013-0253-z
McDermottP.DominguezC.KasdaglisN.RyanM.TrahanI.NelsonA. (2018). Human-machine teaming systems engineering guide. MITRE Corp.
53.
McDermottP. L.ten BrinkR. (2019). Practical guidance for evaluating calibrated trust. In Proceedings of the human factors and ergonomics society annual meeting (Vol. 63, pp. 362–366). Sage Publications. https://doi.org/10.1177/1071181319631379
54.
McKinneyS. M.SieniekM.GodboleV.GodwinJ.AntropovaN.AshrafianH.BackT.ChesusM.CorradoG. S.DarziA.EtemadiM.Garcia-VicenteF.GilbertF. J.Halling-BrownM.HassabisD.JansenS.KarthikesalingamA.KellyC. J.KingD.ShettyS. (2020). International evaluation of an AI system for breast cancer screening. Nature, 577(7788), 89–94. https://doi.org/10.1038/s41586-019-1799-6
55.
MoreyD. A.VellaD. D.RayoM. F.ZelikD. J.MurphyT. B. (2022). Joint activity testing: Towards a Multi-dimensional, high-Resolution evaluation method for human-machine teaming. Proceedings of the Human Factors and Ergonomics Society - Annual Meeting, 66(1), 2214–2219. https://doi.org/10.1177/1071181322661537
56.
MoreyD. A.WalliP.CassidyK. S.TewaniP. K.ReynoldsM. E.MaloneS.JalaeianM.RayoM. F.McGeorgeN. M. (2023). Towards joint activity design heuristics: Essentials for human-machine teaming. Proceedings of the Human Factors and Ergonomics Society - Annual Meeting, 67(1), 21695067231193646. https://doi.org/10.1177/21695067231193646
NaikarN.BradyA.MoyG.KwokH. W. (2023). Designing human-AI systems for complex settings: Ideas from distributed, joint, and self-organising perspectives of sociotechnical systems and cognitive work analysis. Ergonomics (London), 66(11), 1669–1694. https://doi.org/10.1080/00140139.2023.2281898
59.
NaikarN.PearceB. (2003). Analyzing activity for future systems. Proceedings of the Human Factors and Ergonomics Society - Annual Meeting, 47(17), 1928–1932. https://doi.org/10.1177/154193120304701706
60.
National Academies of Sciences, Engineering, and Medicine. (2022). Human-AI teaming: State-of-the-art and research needs. The National Academies Press. https://doi.org/10.17226/26355
61.
NguyenA.YosinskiJ.CluneJ. (2014). Deep neural networks are easily fooled. ArXiv:1412.1897v1.
62.
NormanD. A. (1993). Things that make us smart: Defending human attributes in the age of the machine. Addison Wesley.
63.
O’NeillT.McNeeseN.BarronA.SchelbleB. (2022). Human–autonomy teaming: A review and analysis of the empirical literature. Human Factors, 64(5), 904–938. https://doi.org/10.1177/0018720820960865
64.
ParasuramanR.ManzeyD. (2010). Complacency and bias in human use of automation: An attentional integration. Human Factors, 52(3), 381–410. https://doi.org/10.1177/0018720810376055
65.
ParasuramanR.SheridanT. B.WickensC. D. (2000). A model for types and levels of human interaction with automation. IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans: A Publication of the IEEE Systems, Man, and Cybernetics Society, 30(3), 286–297. https://doi.org/10.1109/3468.844354
66.
ParkinsonS.WardP.WilsonK.MillerJ. (2017). Cyber threats facing autonomous and connected vehicles: Future challenges. IEEE Transactions on Intelligent Transportation Systems, 18(11), 2898–2915. https://doi.org/10.1109/tits.2017.2665968
PhillipsE.SchaeferK. E.BillingsD. R.JentschF.HancockP. A. (2016). Human-animal teams as an analog for future human-robot teams: Influencing design and fostering trust. Journal of Human-Robot Interaction, 5(1), 100–125. https://doi.org/10.5898/jhri.5.1.phillips
69.
RochlinG. I.La PorteT. R.RobertsK. H. (1987). The self-designing high-reliability organization: Aircraft carrier flight operations at sea. Naval War College Review, 40(4), 76–90.
RothE. M.DePassB.HarterJ.ScottR.WamplerJ. (2018). Beyond levels of automation: Developing more detailed guidance for human automation interaction. Proceedings of the Human Factors and Ergonomics Society - Annual Meeting, 62(1), 150–154. https://doi.org/10.1177/1541931218621035
72.
RothE. M.DePassB.ScottR.TruxlerR.SmithS.WamplerJ. (2017). Designing collaborative planning systems: Putting joint cognitive systems principles to practice. In SmithP. J.HoffmanR. R. (Eds). Cognitive systems engineering: The future for a changing world. Taylor & Francis, CRC Press. (247-268).
73.
RothE. M.MalinJ. T.SchreckenghostD. L. (1997). Paradigms for intelligent interface design. In HelanderM.LandauerT.PrabhuP. (Eds.), Handbook of human-computer interaction (2nd ed., pp. 1177–1201). North-Holland.
74.
RyseffJ.NewberryS. J.De BruhlB. (2024). The root causes of failure for artificial intelligence projects and how they can succeed: Avoiding the anti-patterns of AI. RAND Research Report.
75.
SalweiM. E.CarayonP. (2022). A sociotechnical systems framework for the application of artificial intelligence in helath care delivery. Journal of Cognitive Engineering and Decision Making, 16(4), 194–206. https://doi.org/10.1177/15553434221097357
76.
SarkarA. (2023). Enough with “human-AI collaboration”. Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems, 1–8, 415. https://doi.org/10.1145/3544549.3582735
77.
SarterN. B.WoodsD. D. (1995). “From tool to agent”: The evolution of (cockpit) automation and its impact on human-machine coordination. Proceedings of the Human Factors and Ergonomics Society - Annual Meeting, 39(1), 79–83. https://doi.org/10.1177/154193129503900119
78.
ShaoZ.ZhaoR.YuanS.DingM.WangY. (2022). Tracing the evolution of AI in the past decade and forecasting the emerging trends. Expert Systems with Applications, 209, Article 118221. https://doi.org/10.1016/j.eswa.2022.118221
79.
SheridanT. B.VerplankW. L. (1978). Human and computer control of undersea teleoperators. MIT Man-Machines System Laboratory.
ShneidermanB. (2020). Design lessons from AI’s two grand goals: Human emulation and useful applications. IEEE Transactions on Technology and Society, 1(2), 73–82. https://doi.org/10.1109/tts.2020.2992669
82.
ShneidermanB. (2022a). Human-centered AI. Oxford University Press.
SzegedyC.ZarembaW.SutskeverI.BrunaJ.ErhanD.GoodfellowI.FergusR. (2013). Intriguing properties of neural networks. https://arxiv.org/abs/1312.6199
86.
VasconcelosH.JörkeM.Grunde-McLaughlinM.GerstenbergT.BernsteinM. S.KrishnaR. (2023). Explanations can reduce overreliance on AI systems during decision-making. Proc. ACM Human.-Computer Interact, 7(CSCW1), 38. https://doi.org/10.1145/3579605
87.
WoodsD. D. (1985). Cognitive technologies: The design of joint human-machine cognitive systems. AI Magazine, 6(4), 86.
88.
WoodsD. D. (1996). Decomposing automation: Apparent simplicity, real complexity. In ParasuramanR.MoulouaM. (Eds.), Automation and human performance (pp. 3–17). Erlbaum.
WoodsD. D.RothE. M. (1988). Cognitive engineering: Human problem solving with tools. Human Factors: The Journal of the Human Factors and Ergonomics Society, 30(4), 415430. https://doi.org/10.1177/001872088803000404
91.
ZhaoJ.ZhaoW.DengB.WangZ.ZhangF.ZhengW.BurkeA. F.NanJ.LianY. (2024). Autonomous driving system: A comprehensive survey. Expert Systems with Applications, 242, Article 122836. https://doi.org/10.1016/j.eswa.2023.122836
