Challenges and Lessons Learned for Conducting Usability Evaluations in Applied Military Research Environments

Abstract

Conducting usability evaluations in applied military environments is notably different from traditional usability testing, as such environments pose restrictions on when, where, and how testing is conducted. In addition, specialized experience needed from military personnel pose challenges to recruitment and obtaining sufficient time with representative end users. This paper will discuss some challenges and lessons learned for consideration before, during, and after usability assessment data collection. Main recommendations include: recruiting experts qualified to provide feedback that aligns with the purpose of the evaluation; developing a flexible research protocol to allow for the maximum participation possible from any and all involved; and conducting domain research to supplement the researchers’ own knowledge base to minimize reliance on SMEs. Finally, it is important to carefully document not only results, but knowledge and experience gained from specific efforts in order to assist future researchers in conducting work with similar systems.