Streamlining the Design Process: Running Fewer Subjects

Abstract

Recent attention has been focused on making user interface design less costly and more easily incorporated into the product development life cycle. This paper reports an experiment conducted to determine the minimum number of subjects required for a usability test. It replicates work done by Jakob Nielsen and extends it by incorporating problem importance into the curves relating the number of subjects used in an evaluation to the number of usability problems revealed. The basic findings are that (1) with between 4 and 5 subjects, 80% of the usability problems are detected and (2) that additional subjects are less and less likely to reveal new information. Moreover, the correlation between expert judgments of problem importance and likelihood of discovery is significant, suggesting that the most disruptive usability problems are found with the first few subjects. Ramifications for the practice of human factors are discussed as they relate to the type of usability test cycle the practitioner is employing, and the goals of the usability test.