An Empirical Study of Usability Testing: Heuristic Evaluation Vs. User Testing

Abstract

In this study, two different usability-testing methods (Heuristic Evaluation and User Testing) were selected to test the usability of a pre-release version of software searching for Science, Mathematics and Engineering education materials. Our major goal is to compare Heuristic Evaluation and User Testing in terms of efficiency, effectiveness and cost/benefit analysis. We found that Heuristic Evaluation was more efficient than User Testing in finding usability problems (41 vs. 10), while User Testing was more effective than Heuristic Evaluation in finding major problems (70% vs.12%). in general, Heuristic Evaluation appears to be more economic in finding a wide range of usability problems by incurring a low cost in comparison to User Testing. However, User Testing can provide more insightful data from real users such as user's performance and satisfaction.

Get full access to this article

View all access options for this article.

References

Bailey

W. R.

(1996). Human performance engineering: Designing high quality, professional user interfaces for computer products, applications, and systems (3rd ed.). Upper Saddle River, NJ: Prentice Hall PTR.

Chin

J. P.

Diehl

V. A.

Norman

K. L.

(1988). Development of an instrument measuring user satisfaction of the Human-Computer Interface. ACM CHI'88 Proceedings, 213–218.

Lin

X. H.

Choong

Salvendy

(1997). A proposed index of usability: A method for comparing the relative usability of different software systems. Behaviour & Information Technology, 16 (4/5), 267–278.

Nielsen

Molich

(1990). Heuristic evaluation of user interfaces. Proc. ACM CHI'90 Conf, 249–256.

Nielsen

Phillips (1993a). Estimating the relative usability of two interfaces: Heuristic, formal, and empirical methods compared. Proc. ACM CHI'93 Conf, 214–221.

Nielsen

(1993b). Usability Engineering, Boston, MA: Academic Press.

Nielsen

(1994a). Usability inspection methods. Proc. ACM CHI'94, Boston, Massachusetts USA, April 24–28.

Nielsen

(1994b), Jakob Nielsen's Online Writings on HE. Retrieved January 16, 2000, from http://www.useit.com/papers/heuristic/.

Nielsen

(1994c). Enhancing the explanatory power of usability heuristics. CHI'94 Con., 152–155.

10.

Rubin

(1994). Handbook of Usability Testing: How to plan, design, and conduct effective tests. New York: John Wiley & Sons, Inc.

11.

Wang

Caldwell

B.S.

Zhang

(2002). Time delay tolerance in Computer Supported Cooperative Work. Proceeding of 6th International Scientific Conference on Work With Display Unit, 199–201.

12.

Wang

Caldwell

B. S.

Salvendy

(2002). Usability Comparison: Similarity and differences between E-commerce and World Wide Web. Journal of Chinese Industrial Engineering. Accepted for Publication.