New methodologies and quantitative measurements for evaluating human-robot team performance must be developed to achieve effective coordination between teams of humans and unmanned vehicles. The Mixed Initiative Team Performance Assessment System (MITPAS) provides such a comprehensive measurement methodology. MITPAS consists of a methodology, tools and procedures to measure the performance of mixed manned and unmanned teams in both training and real world operational environments. This paper describes MITPAS and the results of an initial experiment conducted to validate the measures and gain insight into the effect of robot competence on operator trust as well as on human-robot team performance.
Get full access to this article
View all access options for this article.
References
1.
AdamsJ. A. (2005). nan-robot interaction design: derstanding user needs and reqirements. Proceedings Human Factors and Ergonomics Society 49th Annual Meeting.
2.
AlbusJ. S. (2002). Metrics and performance: Measures for intelligent unmanned ground vehicles. In MIS Proceedings.
3.
BarnesM.ParasuramanR.CosenzoK. (In press) Adaptive automation for robotic military systems, Technical Report, NATO HFM.
4.
CloughB. T. (2002). Metrics, schmetrics How the heck do you determine a UAV's autonomy anyway? AFB in MIS Proceedings. Air Force Research Lab, Wright Patterson.
5.
CosenzoK.ParasuramanR.NovakA.BarnesM., (2006). Adaptive automation for robotic military systems. ARL Technical Report, ARL-TR-3808.
6.
EndsleyM. R.Direct measurement of Situation Awareness: Validity and use of SAGAT. In EndsleyM. R.GarlandD. J.Situation Areness Analysis and Measurement. Mahwah, NJ: Lawrence Erlbaum Associates.
7.
FallonC. K. (2005). Improving user trust ith a likelihood alarm display. Paper presented at the Proceedings of the 1st International Conference on Augmented Cognition, Las Vegas, NV.
8.
FongT.KaberD.LewisM.ScholtzJ.ShultzA.SteinfeldA. (2004). A common metrics for human-robot interaction. IEEE International Conference on Intelligent Robots and Systems, Sendai, Japan.
9.
FongT.ThorpeC.BaurC. (2002). Collaboration, dialogue and human robot interaction. In proceedings of the 10th International Symposium on Robotic Research, Spinger Veralng.
10.
FoxJ. E.Boehm-DavisD. A. (1998). Effects of age and congestion information accuracy of advanced traveler information systems on user trust and compliance. Transportation Research Record 1 (Safety and nan Performance). Washington, D. C.: National Academy Press43–49.
11.
FreedyA.McDonoughJ. G.FreedyE. T.JacobsR.ThayerS. M.WeltmanG. (2004). A mixed initiative team performance assessment system (MITPAS) for use in training and operational environments. SBIR Phase I Final Report, Perceptronics Solutions Contract No. NC-.
12.
KalphatH. M.StahlJ. (2002). STRICOM's advanced robotics simulation STO: The army solution to robotics M&S. In proceedings of the eleventh conference on computer generated forces ehavioral representation, Orlando, FL.
13.
LeeJ. D.SeeK. A. (2004). Trust in automation: Designing for appropriate reliance. Nan Factors (1), 50–80.
14.
ParasuramanR.GalsterS.SquireP.FurukawaH.MillerC. (2005). A flexible delegation-type interface enhances system performance in human supervision of multiple robots: Empirical studies with RoboFlag. IEEE Transactions on Systems, Man, and Cybernetics. Part A. Systems and nans, 3, 481–493.
15.
ParasuramanR.SheridanT.WickensC. (2000). A model for types and levels of human interaction with automation. IEEE Transactions on Systems, Man, and Cybernetics — Part A: Systems and nans, 0, 286–297.
16.
RoviraE.McGarryK.ParasuramanR. (in press) Effects of imperfect automation on decision-making in a simulated command and control task. nan Factors.
