Critical Incident Data and Their Importance in Remote Usability Evaluation

Abstract

Although lab-based formative evaluation is effective for improving usability of user interfaces, it has limitations. Since most software has a life cycle extending well beyond the first release, the need for usability improvement does not end with deployment. To capture post-deployment usability data, we created the user-reported critical incident method, a method for remote, contemporaneous reporting of critical incidents by users for usability improvement. This method provides us with contextualized critical incident data - critical incident reports plus video and/or textual description of task context. Our studies of this self-reported critical incident method for remote usability evaluation show that users, located in their own working environment and with no background in software engineering or human-computer interaction and with the barest minimum of training, can identify, report, and rate the severity level of their own critical incidents during usage.