Mind the Gaps: How AI Shortcomings and Human Concerns May Disrupt Team Cognition in Human-AI Teams (HATs)

Abstract

As organizations increasingly adopt AI for collaborative work, humans must learn to integrate AI teammates into their workflows. For these human-AI teams (HATs) to function effectively, strong team cognition, encompassing shared understanding, situational awareness, and coordination, is essential. However, unfavorable past experiences, mismatched expectations, and differences in capabilities can make collaboration with AI teammates difficult. To better understand these challenges, we interviewed 30 professionals in high AI-exposure fields to understand their concerns about AI teammates related to team cognition. Participants reported difficulty trusting and relying on AI teammates and expressed fears that overreliance could erode human skills and critical thinking. They also highlighted AI’s limited communication abilities and lack of emotional processing as barriers to effective teamwork. These concerns point to specific behaviors AI teammates must avoid and offer design implications for creating AI teammates that support the effective formation and sustainment of team cognition in HATs.

Keywords

AI teammate human-AI team team cognition human perception

Get full access to this article

View all access options for this article.

References

Aneja

Hoegen

McDuff

Czerwinski

(2019). Designing style matching conversational agents. arXiv preprint arXiv:1910.07514. https://doi.org/10.48550/arXiv.1910.07514

Ayele

Tsoi

(2024). The usage of new AI technologies by college students and its influence on learning and dependence. JSTOR. https://www.jstor.org/stable/pdf/community.37312334.pdf?acceptTC=true&coverpage=false&addFooter=false

Bierhals

Schuster

Kohler

Badke-Schaub

(2007). Shared mental models—linking team cognition and performance. CoDesign, 3(1), 75–94. https://doi.org/10.1080/15710880601170891

Breazeal

(2003). Toward sociable robots. Robotics and Autonomous Systems, 42(3–4), 167–175. https://doi.org/10.1016/S0921-8890(02)00373-1

Braun

Clarke

(2006). Using thematic analysis in psychology. Qualitative Research in Psychology, 3(2), 77–101. https://doi.org/10.1191/1478088706qp063oa

Braun

Clarke

(2019). Reflecting on reflexive thematic analysis. Qualitative Research in Sport, Exercise and Health, 11(4), 589–597. https://doi.org/10.1080/2159676X.2019.1628806

Braun

Clarke

Hayfield

Davey

Jenkinson

(2022). Doing reflexive thematic analysis. In Bager-Charleson

McBeath

(Eds.), Supporting research in counselling and psychotherapy: Qualitative, quantitative, and mixed methods research (pp. 19–38). Springer International Publishing. https://doi.org/10.1007/978-3-031-13942-0_2

Clarke

Braun

(2016). Thematic analysis. The Journal of Positive Psychology, 12(3), 297–298. https://doi.org/10.1080/17439760.2016.1262613

Cannon-Bowers

J. A.

Salas

Converse

S. A.

(1993). Shared mental models in expert team decision making. In Castellan

N. J.

(Ed.), Individual and group decision making: Current issues (pp. 221–246). Lawrence Erlbaum Associates.

10.

Cooke

N. J.

Salas

Cannon-Bowers

J. A.

Stout

R. J.

(2000). Measuring team knowledge. Human Factors, 42(1), 151–173. https://doi.org/10.1518/001872000779656561

11.

Demir

McNeese

N. J.

Cooke

N. J.

(2016). Team communication behaviors of the human-automation teaming. In 2016 IEEE international multi-disciplinary conference on cognitive methods in situation awareness and decision support (CogSIMA) (pp. 28–34). IEEE. https://doi.org/10.1109/COGSIMA.2016.7497782

12.

Demir

McNeese

N. J.

Cooke

N. J.

(2018). The impact of perceived autonomous agents on dynamic team behaviors. IEEE Transactions on Emerging Topics in Computational Intelligence, 2(4), 258–267. https://doi.org/10.1109/TETCI.2018.2829985

13.

Durso

F. T.

Sethumadhavan

. (2008). Situation awareness: Understanding dynamic environments. Human Factors, 50(3), 442–448. https://doi.org/10.1518/001872008X288448

14.

Elish

M. C

. (2019). Moral crumple zones: Cautionary tales in human–robot interaction [Preprint]. Engaging Science, Technology, and Society. Advance onine publication. https://doi.org/10.2139/ssrn.2757236

15.

Elish

M. C

. (2025). Moral crumple zones: Cautionary tales in human–robot interaction. In Robot law: Volume II (pp. 83–105). Edward Elgar Publishing. https://doi.org/10.4337/9781800887305.00010

16.

Endsley

M. R.

Kiris

E. O.

(1995). The out-of-the-loop performance problem and level of control in automation. Human Factors, 37(2), 381–394. https://doi.org/10.1518/001872095779064555

17.

Flathmann

McNeese

N. J.

Schelble

Knijnenburg

Freeman

(2023). Understanding the impact and design of AI teammate etiquette. Human–Computer Interaction, 39(5–6), 444–471. https://doi.org/10.1080/07370024.2023.2189595

18.

Flathmann

Schelble

B. G.

Rosopa

P. J.

McNeese

N. J.

Mallick

Madathil

K. C.

(2023). Examining the impact of varying levels of AI teammate influence on human-AI teams. International Journal of Human-Computer Studies, 177, 1–16. https://doi.org/10.1016/j.ijhcs.2023.103061

19.

Glikson

Woolley

A. W.

(2020). Human trust in artificial intelligence: Review of Empirical Research. Academy of Management Annals, 14(2), 627–660. https://doi.org/10.5465/annals.2018.0057

20.

Hauptman

A. I.

Schelble

B. G.

McNeese

N. J.

Madathil

K. C.

(2023). Adapt and overcome: Perceptions of adaptive autonomous agents for human-AI teaming. Computers in Human Behavior, 138, 107451. https://doi.org/10.1016/j.chb.2022.107451

21.

Klein

(2008). Naturalistic decision making. Human Factors, 50(3), 456–460. https://doi.org/10.1518/001872008X288385

22.

Klein

Feltovich

P. J.

Bradshaw

J. M.

Woods

D. D.

(2005). Common ground and coordination in joint activity. In Rouse

W. B.

Boff

K. R.

(Eds.), Organizational simulation (pp. 139–184). Wiley. https://doi.org/10.1002/0471739448.ch6

23.

Klingbeil

Grützner

Schreck

(2024). Trust and reliance on AI - An experimental study on the extent and costs of overreliance on AI. Computers in Human Behavior, 160, 108352. https://doi.org/10.1016/j.chb.2024.108352

24.

Lancaster

Gilreath

Mallick

McNeese

N. J.

(2024). Evaluating cross-training’s impact on perceived teaming outcomes for human-AI Teams. Proceedings of the human factors and ergonomics society annual meeting, 68(1), 1469–1472. https://doi.org/10.1177/10711813241262033

25.

Lee

J. D.

See

K. A.

(2004). Trust in automation: Designing for appropriate reliance. Human Factors, 46(1), 50–80. https://doi.org/10.1518/hfes.46.1.50_30392

26.

Mallick

Flathmann

Duan

Schelble

B. G.

McNeese

N. J.

(2024). What you say vs what you do: Utilizing positive emotional expressions to relay AI teammate intent within human–AI teams. International Journal of Human-Computer Studies, 192, 103355. https://doi.org/10.1016/j.ijhcs.2024.103355

27.

Mallick

Flathmann

Lancaster

Hauptman

McNeese

Freeman

(2023). The pursuit of happiness: The power and influence of AI teammate emotion in human-AI teamwork. Behaviour & Information Technology, 43(14), 3436–3460. https://doi.org/10.1080/0144929X.2023.2277909

28.

McNeese

N. J.

Cooke

N. J.

(2016). Team cognition as a mechanism for developing collaborative and proactive decision support in remotely piloted aircraft systems. In Schmorrow

Fidopiastis

(Eds.), Foundations of augmented cognition: Neuroergonomics and operational neuroscience: AC 2016 (Vol. 9744, pp. 198–209). Springer. https://doi.org/10.1007/978-3-319-39952-2_20

29.

McNeese

N. J.

Demir

Chiou

E. K.

Cooke

N. J.

(2021). Trust and team performance in human–autonomy teaming. International Journal of Electronic Commerce, 25(1), 51–72. https://doi.org/10.1080/10864415.2021.1846854

30.

McNeese

N. J.

Demir

Cooke

N. J.

Myers

(2018). Teaming With a Synthetic Teammate: Insights into Human-Autonomy Teaming. Human Factors, 60(2), 262–273. https://doi.org/10.1177/0018720817743223

31.

McNeese

N. J.

Flathmann

O’Neill

T. A.

Salas

(2023). Stepping out of the shadow of human-human teaming: Crafting a unique identity for human-autonomy teams. Computers in Human Behavior, 148, 107874. https://doi.org/10.1016/j.chb.2023.107874

32.

McNeese

N. J.

Schelble

B. G.

Canonico

L. B.

Demir

(2021). Who/what is my teammate? Team composition considerations in human–AI teaming. IEEE Transactions on Human-Machine Systems, 51(4), 288–299. https://doi.org/10.1109/THMS.2021.3086018

33.

Musick

O’Neill

T. A.

Schelble

B. G.

McNeese

N. J.

Henke

J. B.

(2021). What happens when humans believe their teammate is an AI? An investigation into humans teaming with autonomy. Computers in Human Behavior, 122, 106852. https://doi.org/10.1016/j.chb.2021.106852

34.

O’Neill

McNeese

Barron

Schelble

(2022). Human–Autonomy Teaming: A Review and Analysis of the Empirical Literature. Human Factors, 64(5), 904-938. https://doi.org/10.1177/0018720820960865

35.

Picard

R. W.

(2000). Toward computers that recognize and respond to user emotion. IBM Systems Journal, 39(3–4), 705–719. https://doi.org/10.1147/sj.393.0705

36.

Ren

Clement

(2024). Augmenting human teams with robots in knowledge work settings: Insights from the literature. ACM Transactions on Human-Robot Interaction, 13(2), Article 20, 34 pages. https://doi.org/10.1145/3649884

37.

Rix

(2022). From tools to teammates: Conceptualizing humans’ perception of machines as teammates with a systematic literature review. Proceedings of the 55th Hawaii International Conference on System Sciences. https://doi.org/10.24251/HICSS.2022.048

38.

Schelble

B. G.

Flathmann

McNeese

N. J.

O’Neill

Pak

Namara

(2022). Investigating the Effects of Perceived Teammate Artificiality on Human Performance and Cognition. International Journal of Human–Computer Interaction, 39(13), 2686–2701. https://doi.org/10.1080/10447318.2022.2085191

39.

Siemon

(2022). Elaborating team roles for artificial intelligence-based teammates in human-AI collaboration. Group Decision and Negotiation, 31(5), 871–912. https://doi.org/10.1007/s10726-022-09792-z

40.

Zhang

McNeese

N. J.

Freeman

Musick

(2021). “An ideal human”: Expectations of AI teammates in human-AI teaming. Proceedings of the ACM on Human-Computer Interaction, 4(CSCW3), 246. https://doi.org/10.1145/3432945