This paper describes the requirements and overall design of a testbed for a human-agent team. A robust, flexible testbed will enable researchers to evaluate the effectiveness of human-agent teaming concepts and issues. In the testbed, MazeWorld, multiple agents play different roles in which tasks are interdependent. Each role can be served by a human or an autonomous agent. Metrics were developed to capture individual and team effectiveness and allow researchers to compare different types of teams and teamwork protocols. Examples of the current task design are presented with the discussion of the future development of the system.
Get full access to this article
View all access options for this article.
References
1.
BartlettC. E.CookeN. J. (2015). Human-robot teaming in urban search and rescue. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, Los Angeles.
2.
BurkeC. S.StaglK. C.SalasE.PierceL.KendallD. (2006). Understanding team adaptation: A conceptual analysis and model. Journal of Applied Psychology, 91(6), 1189-1207. https://doi.org/10.1037/0021-9010.91.6.1189
3.
ButlerJ. K. (1999, June1, 1999). Trust Expectations, Information Sharing, Climate of Trust, and Negotiation Effectiveness and Efficiency. Group & Organization Management, 24(2), 217-238. https://doi.org/10.1177/1059601199242005
4.
ChenJ. Y.BarnesM. J. (2014). Human–agent teaming for multirobot control: A review of human factors issues. IEEE Transactions on Human-Machine Systems, 44(1), 13-29.
5.
GilbertS.SlavinaA.SinatraA. M.BonnerD.JohnstonJ.HolubJ.MacAllisterA.DorneichM.WinerE. (2017). Creating a Team Tutor Using GIFT. International Journal of Artificial Intelligence in Education, 28(2), 286-313. https://doi.org/10.1007/s40593-017-0151-2
6.
JohnsonM.JonkerC.Van RiemsdijkB.FeltovichP. J.BradshawJ. M. (2009). Joint activity testbed: Blocks world for teams (BW4T). International Workshop on Engineering Societies in the Agents World, Heidelberg.
7.
OstranderA.BonnerD.WaltonJ.SlavinaA.OuversonK.KohlA.GilbertS. B.DorneichM. C.SinatraA.WinerE. (2019). Evaluation of an Intelligent Team Tutoring System for a Collaborative Two-Person Problem: Surveillance. Computers in Human Behavior, 104. https://doi.org/https://doi.org/10.1016/j.chb.2019.01.006
8.
OstranderA.GilbertS.DorneichM. (2019). Team Data Analysis Using FATE: Framework for Automated Team Evaluation. In SinatraA. M.DeFalcoJ. A. (Eds.), Proceedings of the Approaches and Challenges in Team Tutoring Workshop (pp. 5-14). Springer. http://ceurws.org/Vol-2501/
9.
ParasuramanR.SheridanT. B.WickensC. D. (2000). A model for types and levels of human interaction with automation. IEEE Transactions on systems, man, and cybernetics-Part A: Systems and Humans, 30(3), 286-297.
10.
SalasE.StaglK. C.BurkeC. S.GoodwinG. F. (2007). Fostering team effectiveness in organizations: Toward an integrative theoretical framework. In ShuartB.SpauldingW.PolandJ. (Eds.), Modeling Complex Systems (vol. 52, pp. 185-244). University of Nebraska Press.
11.
SottilareR. A.BurkeC. S.SalasE.SinatraA. M.JohnstonJ.GilbertS. B. (2018). Designing Adaptive Instruction for Teams: A Meta-Analysis. International Journal of Artificial Intelligence in Education, 28(2), 225-264.
12.
WalliserJ. C.de VisserE. J.WieseE.ShawT. H. (2019). Team Structure and Team Building Improve Human– Machine Teaming With Autonomous Agents. Journal of Cognitive Engineering and Decision Making, 13(4), 258-278.
13.
WaltonJ.GilbertS.WinerE.DorneichM.BonnerD. (2015). Evaluating Distributed Teams with the Team Multiple Errands Test. In Interservice/Industry Training, Simulation, and Education Conference (I/ITSEC) (pp. 1-12, Paper 15264).
