This study investigated whether humans can be efficiently trained in human-agent teams (HATs) teamwork competencies to improve HAT collaboration. In HATs, humans and artificially intelligent (AI) agents collaborate on shared tasks, which requires teamwork. However, human and agent approaches to teamwork differ, posing challenges in HATs. These challenges raise the need to train humans to develop teamwork competencies that they can effectively apply in HAT settings. The cooperative video game, KeyWe, was used as a testbed, in which human participants completed tasks with a scripted agent. A HAT training intervention that took less than 30 minutes was developed to train humans on seven teamwork competencies. The training was not associated with the KeyWe game task itself. Half of the participants received the training, and half did not. Participants who received the training delegated a higher percentage of tasks to the agent and more often assigned tasks to the agent by defining strategies than participants who did not receive teamwork training. Trained teams demonstrated resilience by achieving higher task performance when the game difficulty increased. This study demonstrated that training humans to develop teamwork competencies, independent from task training, can enhance collaboration and performance in HATs.
AmreshA.CookeN.FouseA. (2023, September). A Minecraft based simulated task environment for human AI teaming. In Proceedings of the 23rd ACM International Conference on Intelligent Virtual Agents (IVA (2023). Würzburg, Germany: ACM. https://doi.org/10.1145/3570945.3607305
3.
AsialaL.McCarthyJ. E.HuangL. (2023). Improving the State of the Art for Training Human-AI Teams: Technical Report# 3--Analysis of Testbed Alternatives. arXiv preprint arXiv:2309.03213.
4.
BaekI. C.HaT. G.ParkT. H.KimK. J. (2022). Toward cooperative level generation in multiplayer games: A user study in overcooked. In: IEEE Conference on Computatonal Intelligence and Games, CIG, Beijing, China, 21-24 August 2022, 276–283. https://doi.org/10.1109/CoG51982.2022.9893581
5.
BonnerD.GilbertS. B.DorneichM. C.BurkeS.WaltonJ.RayC.WinerE. (2014). Taxonomy of Teams, Team Tasks, and Tutors. Industrial and Manufacturing Systems Engineering Conference Proceedings, 189–198. https://dr.lib.iastate.edu/handle/20.500.12876/44303.
6.
BonnerD.OuversonK.GilbertS.SinatraA. M.DorneichM. C.WinerE.SlavinaA.MacAllisterA.KohlA. (2017, October). Operationalizing the C’s of teamwork in an intelligent tutoring system. In Proceedings of the Human Factors and Ergonomics Society 61st Annual Meeting (61, pp. 745–749). Austin, Texas, USA: Sage Publications.
7.
BrowderD. M.SchoenS. F.LentzF. E. (1986). Learning to learn through observation. The Journal of Special Education, 20(4), 447–461. https://doi.org/10.1177/002246698602000406
8.
BrunerM. W.HallJ.CôtéJ. (2011). Influence of sport type and interdependence on the developmental experiences of youth male athletes. European Journal of Sport Science, 11(2), 131–142. https://doi.org/10.1080/17461391.2010.499969
9.
BurkeC. S.StaglK. C.KleinC.GoodwinG. F.SalasE.HalpinS. M. (2006). What type of leadership behaviors are functional in teams? A meta-analysis. The Leadership Quarterly, 17(3), 288–307. https://doi.org/10.1016/j.leaqua.2006.02.007
CookeN. J.GormanJ. C.MyersC. W.DuranJ. L. (2013). Interactive team cognition. Cognitive Science, 37(2), 255–285. https://doi.org/10.1111/cogs.12009
12.
CummingJ.ClarkS. E.Ste-MarieD. M.McCullaghP.HallC. (2005). The functions of observational learning questionnaire (FOLQ). Psychology of Sport and Exercise, 6(5), 517–537. https://doi.org/10.1016/j.psychsport.2004.03.006
13.
DavisF. D.YiM. Y. (2004). Improving computer skill training: Behavior modeling, symbolic mental rehearsal, and the role of knowledge structures. Journal of Applied Psychology, 89(3), 509–523. https://doi.org/10.1037/0021-9010.89.3.509
14.
DemirM.CookeN. J.AmazeenP. G. (2018). A conceptual model of team dynamical behaviors and performance in human-autonomy teaming. Cognitive Systems Research, 52, 497–507.
15.
DemirM.LikensA. D.CookeN. J.AmazeenP. G.McNeeseN. J. (2019). Team coordination and effectiveness in human-autonomy teaming. IEEE Transactions on Human-Machine Systems, 49(2), 150–159. https://doi.org/10.1109/THMS.2018.2877482
16.
DemirM.McNeeseN. J.CookeN. J. (2016, March). Team communication behaviors of the human-automation teaming. In 2016 IEEE international multi-disciplinary conference on cognitive methods in situation awareness and decision support (CogSIMA) (pp. 28–34). IEEE.
17.
DemirM.McNeeseN. J.CookeN. J. (2017). Team situation awareness within the context of human-autonomy teaming. Cognitive Systems Research, 46, 3–12. https://doi.org/10.1016/j.cogsys.2016.11.003
18.
DorneichM. C.GilbertS.FrancisR. F.TalyatM.DidicE. (2023). Team skill metrics that span human-human and human-agent teams: An initial assessment. Proceedings of the Human Factors and Ergonomics Society, 67(1), 123–130. https://doi.org/10.1177/21695067231192892
19.
EndsleyM. R. (2023). Supporting human-AI teams: Transparency, explainability, and situation awareness. Computers in Human Behavior, 140, 107574. https://doi.org/10.1016/j.chb.2022.107574
20.
FarahY. A.DorneichM. C. (2022). Development of a gamified cooperative teamwork environment using an event-based approach. In Extended abstracts of the 2022 annual symposium on computer-human interaction in play, Bremen, Germany, 2 - 5 November 2022, pp. 196–202.
21.
FarahY. A.DorneichM. C.GilbertS. B. (2022, October). Evaluating team metrics in cooperative video games. In Proceedings of the Human Factors and Ergonomics Society 66th Annual Meeting (66, pp. 70–74). Atlanta, Georgia, USA: Sage Publications.
22.
FernandoN.McAdamT.YoungsonG.McKenzieH.ClelandJ.YuleS. (2007). Undergraduate medical students’ perceptions and expectations of theatre-based learning: How can we improve the student learning experience?The Surgeon, 5(5), 271–274. https://doi.org/10.1016/s1479-666x(07)80024-2
23.
FlathmannC.SchelbleB. G.GaleanoA. (2024, September). Empirical impacts of independent and collaborative training on task performance and improvement in human-AI teams. In Proceedings of the Human Factors and Ergonomics Society 68th Annual Meeting (68, pp. 1447–1453). Phoenix, Arizona, USA: Sage Publications.
24.
FlathmannC.SchelbleB. G.McNeeseN. J.KnijnenburgB.GramopadhyeA. K.Chalil MadathilK. (2024). The purposeful presentation of ai teammates: Impacts on human acceptance and perception. International Journal of Human-Computer Interaction, 40(20), 6510–6527. https://doi.org/10.1080/10447318.2023.2254984
25.
FrancisR. F. (2023). Using team gameplay to develop observable metrics for teaming behavior measurement. Master's thesis. Iowa State University.
GrimmD.DemirM.GormanJ. C.CookeN. J. (2018). Systems level evaluation of resilience in human-autonomy teaming under degraded conditions. In 2018 resilience week (RWS) (pp. 124–130). IEEE.
28.
GrossmanR.SalasE.PavlasD.RosenM. A. (2013). Using instructional features to enhance demonstration-based training in management education. Management Learning & Education, 12(2), 219–243. Journal of Educational Psychology, 93: 377–389. https://doi.org/10.5465/amle.20guideinmultimedialearning
29.
HarrisJ.HancockM. (2019, May 4–9). To asymmetry and beyond! Improving social connectedness by increasing designed interdependence in cooperative play. Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems,Glasgow, Scotland, UK. ACM. https://doi.org/10.1145/3290605.3300239
30.
Harris-WatsonA. M.LarsonL. E.LauharatanahirunN.DeChurchL. A.ContractorN. S. (2023). Social perception in Human-AI teams: Warmth and competence predict receptivity to AI teammates. Computers in Human Behavior, 145, 107765. https://doi.org/10.1016/j.chb.2023.107765
31.
HartA. G.StavelandL. E. (1988). Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research. In: Human mental workload. Elsevier.
32.
IlgenD. R.HollenbeckJ. R.JohnsonM.JundtD. (2005). Teams in organizations: From input-process-output models to IMOI models. Annual Review of Psychology, 56, 517–543. https://doi.org/10.1146/annurev.psych.56.091103.070250
33.
JohnsonC. J.DemirM.McNeeseN. J.GormanJ. C.WolffA. T.CookeN. J. (2023). The impact of training on human–autonomy team communications and trust calibration. Human factors, 65(7), 1554–1570.
34.
KelleyJ. F. (1984). An iterative design methodology for user-friendly natural language office information applications. ACM Transactions on Information Systems, 2(1), 26–41. https://doi.org/10.1145/357417.357420
35.
KendallD. L.SalasE. (2004). Measuring team performance: A review of current methods and consideration of future needs. In NessJ. M.TepeV.RitzerD. (Eds.), Advances in human performance and cognitive engineering research (5, pp. 307–326). Elsevier Science/JAI Press.
36.
KleinG.FeltovichP. J.BradshawJ. M.WoodsD. D. (2004). Common ground and coordination in joint activity. In Organizational Simulation (pp. 139–184). Germany: John Wiley & Sons, Incorporated.
37.
KobayashiK. (2005). What limits the encoding effect of note-taking? A meta-analytic examination. Contemporary Educational Psychology, 30(2), 242–262. https://doi.org/10.1016/j.cedpsych.2004.10.001
38.
LancasterC. M.DuanW.MallickR.McNeeseN. J. (2025). Human-centered team training for Human-AI teams: From training with AI tools to training for AI teammates. Proceedings of the ACM on Human-Computer Interaction, 9(2), 1–38. https://doi.org/10.1145/3710998
39.
LyonsJ. B.SycaraK.LewisM.CapiolaA. (2021). Human–autonomy teaming: Definitions, debates, and directions. In: Frontiers in Psychology (12, 589585). Frontiers Media S.A. https://doi.org/10.3389/fpsyg.2021.589585
40.
MarksM. A.MathieuJ. E.ZaccaroS. J. (2001). A temporally based framework and taxonomy of team processes. Source: Academy of Management Review, 26(Issue 3), 356. https://www.jstor.org/stable/259182?seq=1&cid=pdf-
41.
MathieuJ. E.HollenbeckJ. R.van KnippenbergD.IlgenD. R. (2017). A century of work teams in the Journal of Applied Psychology. Journal of Applied Psychology, 102(3), 452–467. https://doi.org/10.1037/apl0000128
42.
MautoneP. D.MayerR. E. (2001). Signaling as a cognitive guide in multimedia learning. Journal of educational Psychology, 93(2), 377–389.
43.
McNeeseN. J.DemirM.CookeN. J.MyersC. (2018). Teaming with a synthetic teammate: Insights into human-autonomy teaming. Human Factors, 60(2), 262–273. https://doi.org/10.1177/0018720817743223
44.
McNeeseN. J.FlathmannC.O’NeillT. A.SalasE. (2023). Stepping out of the shadow of human-human teaming: Crafting a unique identity for human-autonomy teams. Computers in Human Behavior, 148, 107874. https://doi.org/10.1016/j.chb.2023.107874
45.
MeyerJ.LeeJ. D. (2013). Trust, reliance, and compliance. In The Oxford handbook of cognitive engineering (pp. 109–124). Oxford, UK: Oxford University Press.
46.
MorschheuserB.MaedcheA.WalterD. (2017, February). Designing cooperative gamification: Conceptualization and prototypical implementation. In Proceedings of the 2017 acm conference on computer supported cooperative work and social computing (pp. 2410–2421). https://doi.org/10.1145/2998181.2998272
47.
MusickG.O’NeillT. A.SchelbleB. G.McNeeseN. J.HenkeJ. B. (2021). What happens when humans believe their teammate is an AI? An investigation into humans teaming with autonomy. Computers in Human Behavior, 122, 106852. https://doi.org/10.1016/j.chb.2021.106852
48.
NarjaikaewP.EmaratN.CowieB. (2009). The effect of guided note taking during lectures on Thai university students’ understanding of electromagnetism. Research in Science & Technological Education, 27(1), 75–94. https://doi.org/10.1080/02635140802658917
49.
National Academies of Sciences, Engineering, and Medicine. (2022). Human-AI teaming: state-Of-The-art and research needs. The National Academies Press.
50.
NguyenD.CarmodyK.WildmanJ.CarrollM. (2022, November). Developing a dynamic testbed: Designing for trust in complex human agent teams. In 2022 IEEE 3rd International Conference on Human-Machine Systems (ICHMS) (pp. 1-4). IEEE. https://doi.org/10.1109/ICHMS56717.2022.9980750
51.
NikolaidisS.LasotaP.RamakrishnanR.ShahJ. (2015). Improved human-robot team performance through cross-training, an approach inspired by human team training practices. The International Journal of Robotics Research, 34(14), 1711–1730. https://doi.org/10.1177/0278364915609673
52.
O’NeillT. A.FlathmannC.McNeeseN. J.SalasE. (2023). 21st century teaming and beyond: Advances in human-autonomy teamwork. Computers in Human Behavior, 147, 107865. https://doi.org/10.1016/j.chb.2023.107865
53.
O’neillT.McNeeseN.BarronA.SchelbleB. (2022). Human–autonomy teaming: A review and analysis of the empirical literature. Human factors, 64(5), 904–938. https://doi.org/10.1177/0018720820960865
54.
OstranderA.BonnerD.WaltonJ.SlavinaA.OuversonK.KohlA.GilbertS. B.DorneichM. C.SinatraA.WinerE. (2020). Evaluation of an intelligent team tutoring system for a collaborative two-person problem: Surveillance. Computers in Human Behavior, 104, 105873. https://doi.org/10.1016/j.chb.2019.01.006
55.
ParasuramanR.SheridanT. B.WickensC. D. (2000). A model for types and levels of human interaction with automation. IEEE Transactions on systems, man, and cybernetics-Part A. Systems and Humans, 30(3), 286–297.
56.
RosenM. A.BedwellW. L.WildmanJ. L.FritzscheB. A.SalasE.BurkeC. S. (2011). Managing adaptive performance in teams: Guiding principles and behavioral markers for measurement. Human Resource Management Review, 21(2), 107–122. https://doi.org/10.1016/j.hrmr.2010.09.003
57.
RousseauV.AubéC.SavoieA. (2006). Teamwork behaviors: A review and an integration of frameworks. Small Group Research, 37(Issue 5), 540–570. https://doi.org/10.1177/1046496406293125
58.
SaavedraR.EarleyP. C.Van DyneL. (1993). Complex interdependence in task-performing groups. Journal of Applied Psychology, 78(1), 61–72. https://doi.org/10.1037//0021-9010.78.1.61
SalasE.CookeN. J.RosenM. A. (2008). On teams, teamwork, and team performance: Discoveries and developments. Human Factors, 50(3), 540–547. https://doi.org/10.1518/001872008X288457
61.
SalasE.ReyesD. L.McDanielS. H. (2018). The science of teamwork: Progress, reflections, and the road ahead. American Psychologist, 73(4), 593–600. https://doi.org/10.1177/1046496405277134
62.
SalasE.ReyesD. L.WoodsA. L. (2017). The Assessment of Team Performance: Observations and Needs. In Innovative Assessment of Collaboration. Springer, Cham. https://doi.org/10.1007/978-3-319-33261-1_2
Seif El-NasrM.AghabeigiB.MilamD.ErfaniM.LamemanB.MaygoliH.MahS. (2010, April). Understanding and evaluating cooperative games. In Proceedings of the SIGCHI conference on human factors in computing systems (pp. 253–262).
65.
SepichN.DorneichM. C.GilbertS. (2021). Human-agent team game analysis framework: Case studies. In: Proceedings of the human factors and ergonomics society annual meeting, Baltimore, Maryland, USA, 3-8 October 2021, 65(1), pp. 1146–1150, Sage Publications. https://doi.org/10.1177/1071181321651188
66.
SessaV. I. (1996). Using perspective taking to manage conflict and affect in teams. The Journal of applied behavioral science, 32(1), 101–115.
67.
SottilareR. A.Shawn BurkeC.SalasE.SinatraA. M.JohnstonJ. H.GilbertS. B. (2018). Designing adaptive instruction for teams: A meta-analysis. International Journal of Artificial Intelligence in Education, 28(2), 225–264.
StowersK.BradyL. L.MacLellanC.WohleberR.SalasE. (2021). Improving teamwork competencies in human-machine teams: Perspectives from team science. Frontiers in Psychology, 12, 590290. https://doi.org/10.3389/fpsyg.2021.590290
70.
TengC. I.DennisA. R.DennisA. S. (2023). Avatar-mediated communication and social identification. Journal of Management Information Systems, 40(4), 1171–1201. https://doi.org/10.1080/07421222.2023.2267320
71.
TokadlıG.OuversonK.MeuselC.GarciaA.GilbertS. B.DorneichM. C. (2018). An analysis of video games using the dimensions of human-agent interaction. In Proceedings of the human factors and ergonomics society - annual meeting, Philadelphia, Pennsylvania, USA, 1-5 October 2018, 62(1), pp. 716–720, Sage Publications. https://doi.org/10.1177/1541931218621163
72.
WalliserJ. C.de VisserE. J.WieseE.ShawT. H. (2019). Team structure and team building improve human–machine teaming with autonomous agents. Journal of Cognitive Engineering and Decision Making, 13(4), 258–278. https://doi.org/10.1177/1555343419867563
73.
WieseC. W.ShufflerM. L.SalasE. (2015). Teamwork and team performance measurement. In International encyclopedia of the social & behavioral sciences (Second Edition, pp. 96–103). Elsevier Inc. https://doi.org/10.1016/B978-0-08-097086-8.22017-5
74.
WilsonG. M.FarahY.FurukawaV.TanA.DorneichM. C.GilbertS. B. (2022, October). Comparing team performance constructs involved in ground combat and a ground combat simulator. In Proceedings of the Human Factors and Ergonomics Society 66th Annual Meeting (66, pp. 407–411). Atlanta, Georgia, USA: Sage Publications.
75.
WynneK. T.LyonsJ. B. (2019, June). Autonomous agent teammate-likeness: Scale development and validation. In International Conference on Human-Computer Interaction (pp. 199–213). Cham: Springer International Publishing. https://doi.org/10.1007/978-3-030-21565-1_13
76.
ZhangR. (2023). Structuring AI Teammate Communication: An Exploration of AI’s Communication Strategies in Human-AI. Teams (Doctoral dissertation, Clemson University.
77.
ZhangR.DuanW.FlathmannC.McneeseN.FreemanG.WilliamsA. (2023). Investigating AI teammate communication strategies and their impact in human-AI teams for effective teamwork. Proceedings of the ACM on Human-Computer Interaction, 7(CSCW2), 1–31. https://doi.org/10.1145/3610072
78.
ZhangR.McNeeseN. J.FreemanG.MusickG. (2021). “An ideal human”: Expectations of AI teammates in Human-AI teaming. Proceedings of the ACM on Human-Computer Interaction, 4(CSCW3), 1–25. https://doi.org/10.1145/3432945
