The Effectiveness of Usability Evaluation Methods: Determining the Appropriate Criteria

Abstract

Because there are various measures for comparing methods, researchers in human-computer interaction generally find it difficult to make solid conclusions about a particular method. We consider some candidate measures (e.g., thoroughness, validity, reliability) of effectiveness and then provide a summary of studies that have compared usability evaluation methods (UEMs) using one or more of these measures. We find that studies do not always provide the appropriate descriptive statistics to make solid conclusions, especially in terms of validity. In addition, studies do not always compare UEMs to a standard yardstick such as end-user testing to establish an appropriate validity score. Finally, we provide some possible ways to address criterion deficiency and criterion contamination; two important considerations for researchers attempting to optimize the balance between ultimate and actual criteria.

Get full access to this article

View all access options for this article.

References

Cuomo

D. L.

(1994). A method for assessing the usability of graphical, direct-manipulation style interfaces. International Journal of Human-Computer Interaction, 6(3), 275–297.

Cuomo

D. L.

Bowen

C. D.

(1992). Stages of user activity model as a basis for user-system interface evaluations. In Proceedings of the Human Factors Society 36th Annual Meeting (pp. 1254–1258). Santa Monica, CA: Human Factors and Ergonomics Society.

Desurvire

H. W.

Kondziela

J. M.

Atwood

M. E.

(1992). What is gained and lost when using evaluation methods other than empirical testing. In Monk

Diaper

Harrison

M. D.

(Eds.), People and computers VII (pp. 89–102. Cambridge, UK: Cambridge University Press.

Desurvire

H. W.

Thomas

J. C.

(1993). Enhancing the performance of interface evaluators. In Proceedings of the Human Factors and Ergonomics Society 37th Annual Meeting (pp. 1132–1136). Santa Monica, CA: Human Factors and Ergonomics Society.

Doubleday

Ryan

Springett

Sutcliffe

(1997). A comparison of usability techniques for evaluating design. In Designing Interactive Systems (DIS ′97) Conference Proceedings (pp. 101–110). New York: ACM Press.

Dutt

Johnson

(1994). Evaluating evaluation methods. In Cockton

Draper

S. W.

Weir

G. R. S.

(Eds.), People and computers IX (pp. 109–121. Cambridge, UK: Cambridge University Press.

Goldstein

I. L.

(1993). Training in organizations: Needs assessment, development, and evaluation. (3rd ed.). Pacific Grove, CA: Brooks/Cole.

Jeffries

Miller

J. R.

Wharton

Uyeda

K. M.

(1991). User interface evaluation in the real world: A comparison of four techniques. In CHI ′91 Conference Proceedings (pp. 119–124). New York: ACM Press.

John

B. E.

Marks

S. J.

(1997). Tracking the effectiveness of usability evaluation methods. Behaviour and Information Technology, 16, 188–202.

10.

John

B. E.

Mashyna

M. M.

(1997). Evaluating a multimedia authoring tool. Journal of the American Society for Information Science, 48(11), 1004–1022.

11.

Karat

Campbell

R. L.

Fiegel

(1992). Comparison of empirical testing and walkthrough methods in user interface evaluation. In CHI ′92 Conference Proceedings (pp. 397–404). New York: ACM Press.

12.

Lund

A. M.

(1998). The need for a standardized set of usability metrics. In Proceedings of the Human Factors and Ergonomics Society 42nd Annual Meeting (pp. 688–691). Santa Monica, CA: Human Factors and Ergonomics Society.

13.

Nielsen

(1990). Evaluating the thinking aloud technique for use by computer scientists. In Hartson

H. R.

Hix

(Eds.), Advances in human-computer interaction (Vol. 3, pp. 69–82). Norwood, NJ: Ablex.

14.

Nielsen

(1992). Finding usability problems through heuristic evaluation. In CHI ′92 Conference Proceedings (pp. 373–380). New York: ACM Press.

15.

Nielsen

Molich

(1990). Heuristic evaluation of user interfaces. In CHI ′90 Conference Proceedings (pp. 249–256). New York: ACM Press.

16.

Sears

(1997). Heuristic walkthroughs: Finding the problems without the noise. International Journal of Human-Computer Interaction, 9(3), 213–234.

17.

Virzi

Sorce

Herbert

L. B.

(1993). A comparison of three usability evaluation methods: Heuristic, think-aloud, and performance testing. In Proceedings of the Human Factors and Ergonomics Society 36th Annual Meeting (pp. 309–313). Santa Monica, CA: Human Factors and Ergonomics Society.

18.

Virzi

R. A.

(1990). Streamlining the design process: Running fewer subjects. In Proceedings of the Human Factors and Ergonomics Society 34th Annual Meeting (pp. 291–294). Santa Monica, CA: Human Factors and Ergonomics Society.

19.

Virzi

R. A.

(1992). Refining the test phase of usability evaluation: How many subjects is enough? Human Factors, 34(4), 457–468.