A testbed has been developed to study multiple constructs of teamwork in a parameterized environment. MazeWorld is a testbed to learn and automate the collection of objective team metrics based on observable behaviors. Quantifying teamwork plays an important role in future agent cooperation. Challenges include differentiating individual skills from skills of the team to not only understand the team as an entity but each of the participating members. MazeWorld is a Unity-based cooperative game that involves three distinct roles working interdependently to achieve a goal. A series of trials were run in MazeWorld to test proposed observable measures of teaming constructs. Teams performed better as the trials progressed, and differences were found between teams. This effort serves as a proof of concept that MazeWorld can explore team behavior using automated team metrics to inform future work.
Get full access to this article
View all access options for this article.
References
1.
BarrickM. R.StewartG. L.NeubertM. J.MountM. K. (1998). Relating member ability and personality to work-team processes and team effectiveness. Journal of Applied Psychology, 83(3), 377–391. https://doi.org/10.1037/0021-9010.83.3.377
2.
BartlettC. E.CookeN. J. (2015). Human-Robot Teaming in Urban Search and Rescue. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 59(1), 250–254. https://doi.org/10.1177/1541931215591051
3.
BreuerC.HüffmeierJ.HibbenF.HertelG. (2019). Trust in teams: A taxonomy of perceived trustworthiness factors and risk-taking behaviors in face-to-face and virtual teams. Human Relations, 73(1), 3–34. https://doi.org/10.1177/0018726718818721
4.
Cannon-BowersJ. A.TannenbaumS. I.CavanahE.FordZ.JasperA.StonewallJ.GilbertS. B.DorneichM. (2020). Creating Metrics for Human-Agent Teams. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 64(1), 349–353. https://doi.org/10.1177/1071181320641079
5.
ChenJ. Y. C.BarnesM. J. (2014). Human–Agent Teaming for Multirobot Control: A Review of Human Factors Issues. IEEE Transactions on Human-Machine Systems, 44(1), 13–29. https://doi.org/10.1109/thms.2013.2293535
6.
CohenJ. (1988). Statistical power analysis for the behavioral sciences Lawrence Earlbaum Associates. Hillsdale, NJ, 20-26.
7.
CookeN. J.ShopeS. M. (2004). Synthetic task environments for teams: CERTT’s UAV-STE. In Handbook of human factors and ergonomics methods (pp. 476-483). CRC Press.
8.
DayE. A.ArthurW.Jr.GettmanD. (2001). Knowledge structures and the acquisition of a complex skill. Journal of Applied Psychology, 86, 1022–1033
9.
DeliceF.RousseauM.FeitosaJ. (2019). Advancing Teams Research: What, When, and How to Measure Team Dynamics Over Time. Frontiers in Psychology, 10. https://doi.org/10.3389/fpsyg.2019.01324
GelmanA. (2013). Commentary: P values and statistical practice. Epidemiology, 24(1), 69-72.
12.
JohnsonM.BradshawJ. M.DuranD.VignatiM.FeltovichP. J.JonkerC.Birna Van RiemsdijkM. (2009). RT4T: A Reconfigurable Testbed for Joint Human-Agent-Robot Teamwork.
13.
KozlowskiS. W. J. (2007). Team Learning, Development, and Adaptation [E-book]. In BellB. S. (Ed.), Work Group Learning (1st ed., pp. 1–30). Taylor & Francis.
14.
MarksM. A.MathieuJ. E.ZaccaroS. J. (2001). A Temporally Based Framework and Taxonomy of Team Processes. Academy of Management Review, 26(3), 356–376. https://doi.org/10.5465/amr.2001.4845785
15.
MorschheuserB.RiarM.HamariJ.MaedcheA. (2017). How games induce cooperation? A study on the relationship between game features and we-intentions in an augmented reality game. Computers in Human Behavior, 77, 169–183. https://doi.org/10.1016/j.chb.2017.08.026
NassC.FoggB.MoonY. (1996). Can computers be teammates?International Journal of Human-Computer Studies, 45(6), 669–678. https://doi.org/10.1006/ijhc.1996.0073
18.
O’NeillT.McNeeseN.BarronA.SchelbleB. (2020). Human– Autonomy Teaming: A Review and Analysis of the Empirical Literature. Human Factors: The Journal of the Human Factors and Ergonomics Society, 001872082096086. https://doi.org/10.1177/0018720820960865
19.
OstranderA.GilbertS.DorneichM. (2019). Team Data Analysis Using FATE: Framework for Automated Team Evaluation. In SinatraA. M.DeFalcoJ. A. (Eds.), Proceedings of the Approaches and Challenges in Team Tutoring Workshop (pp. 5-14). Springer. http://ceur-ws.org/Vol2501/
20.
ReeM. J.CarrettaT. R.TeachoutM. S. (1995). Role of ability and prior job knowledge in complex training performance. Journal of Applied Psychology, 80,721–730.
21.
SalasE.CookeN. J.RosenM. A. (2008). On Teams, Teamwork, and Team Performance: Discoveries and Developments. Human Factors: The Journal of the Human Factors and Ergonomics Society, 50(3), 540–547. https://doi.org/10.1518/001872008x288457
SalasE.StaglK. C.BurkeC. S.GoodwinG. F. (2007). Fostering team effectiveness in organizations. Toward an integrative theoretical framework of team performance. Modeling Complex Systems. Nebraska Symposium on Motivation, 6, 185–243. https://doi.org/10.1016/s14759144(07)06004-3
24.
SchmidtF. L.HunterJ. E. (1992). Development of a causal model of processes determining job performance. Current Directions in Psychological Science, 1,89–92.
25.
SottilareR. A.Shawn BurkeC.SalasE.SinatraA. M.JohnstonJ. H.GilbertS. B. (2017). Designing Adaptive Instruction for Teams: a Meta-Analysis. International Journal of Artificial Intelligence in Education, 28(2), 225–264. https://doi.org/10.1007/s40593-017-0146-z
WiltshireT. J.ButnerJ. E.FioreS. M. (2017). Problem-Solving Phase Transitions During Team Collaboration. Cognitive Science, 42(1), 129–167. https://doi.org/10.1111/cogs.12482
28.
WaltonJ.GilbertS.WinerE.DorneichM.BonnerD. (2015). Evaluating Distributed Teams with the Team Multiple Errands Test. In Interservice/Industry Training, Simulation, and Education Conference (I/ITSEC) (pp. 1-12, Paper 15264).
