A Comparison of Three Usability Evaluation Methods: Heuristic,Think-Aloud,and Performance Testing

Abstract

A high-fidelity prototype of an extended voice mail application was created. We tested it using three distinct usability testing paradigms so that we could compare the quantity and quality of the information obtained using each. The three methods employed were (1) heuristic evaluation, in which usability experts critique the user interface, (2) think-aloud testing, in which naive subjects comment on the system as they use it, and (3) performance testing, in which task completion times and error rates are collected as naive subjects interact with the system. The three testing methodologies were roughly equivalent in their ability to detect a core set of usability problems on a per evaluator basis. However, the heuristic and think-aloud evaluations were generally more sensitive, uncovering a broader array of problems in the user interface. Implications of these findings are discussed in terms of the costs of doing the evaluations and in light of other work on this topic.

Get full access to this article

View all access options for this article.

References

Bailey

Allan

Raiello

(1992) Usability testing vs. heuristic evaluation: A head-to-head comparison. In Proc. of the Human Factors Society 36th Annual Meeting, pp. 409–412. Human Factors and Ergonomics Society, Monterey, CA.

Desurvire

Lawrence

Atwood

(1991) Empiricism versus judgment: Comparing user interface evaluation methods on a new telephone-based interface. SIGCHI Bulletin, 23, pp. 58–59.

Karat

(1993) Personal communication, February 15.

Karat

Campbell

Fiegel

(1992) Comparison of empirical testing and walkthrough methods in user interface evaluation. In Proc. of ACM CHI′92 Conference on Human Factors in Computing Systems pp. 397–404, ACM, New York.

Landauer

T. K.

(1988) Research Methods in Human Computer Interaction. In Helander

(Ed.) The Handbook of Human Computer Interaction, Elsevier Press: NY.

Nielsen

(1992) Finding usability problems through heuristic evaluation. In Proc. of ACM CHI′92 Conference on Human Factors in Computing Systems pp. 373–380, ACM, New York.

Nielsen

Molich

(1990) Heuristic evaluation of user interfaces. In Proc. of ACM CHI′90 Conference on Human Factors in Computing Systems, pp. 249–256, ACM, New York.

Virzi

(1992) Refining the test phase of usability evaluation: How many subjects is enough?, Human Factors, 34, pp 457–468.

Wharton

Jeffries

Miller

J. R.

Uyeda

K. M.

, (1991). User interface evaluation in the real world: a comparison of four techniques. In Proc. of ACM CHI′91 Conference on Human Factors in Computing Systems, pp. 119–124, ACM, New York.

10.

Wright

Monk

(1991) A cost-effective evaluation method for use by designers. Intl. J. of Man-Machine Studies, 35, pp. 891–912.