Testing a Framework for Reliable Classification of Usability Problems

Abstract

The User Action Framework (UAF) is a knowledge base of usability issues and concepts structured to provide a framework and method for classifying usability problems identified during usability evaluation. The UAF is essentially a hierarchical structure of usability attributes that users traverse as a decision structure, selecting the most appropriate classification category and sub-category at each level of the hierarchy. The cumulative set of category choices along the classification path is taken as a sequence of usability attributes that determines a complete classification description of the usability problem in question. The UAF itself has been the subject of usability evaluation and is the product of an extensive, iterative design process. In this paper we report on the reliability of the UAF by measuring the agreement of 10 experienced usability practitioners as they classify 15 different usability problems.

Get full access to this article

View all access options for this article.

References

Brooks

(1994). Adding value to usability testing. In Nielsen

Mack

R. L.

(Eds.), Usability inspection methods (pp. 255–271). New York: John Wiley & Sons.

Carroll

J. M.

Kellogg

W. A.

Rosson

M. B.

(1991). The task-artifact cycle. In Carroll

J. M.

(Ed.), Designing interaction: Psychology at the human-computer interface (pp. 74–102). Cambridge, UK: Cambridge University Press.

Cohen

(1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1), 37–46.

Desurvire

H. W.

(1994). Faster, Cheaper! Are usability inspection methods as effective as empirical testing? In Nielsen

Mack

R. L.

(Eds.), Usability inspection methods (pp. 173–202). New York: John Wiley & Sons.

Fleiss

J. L.

(1971). Measuring nominal scale agreement among many raters. Psychological Bulletin, 76(5), 378–382.

Jeffries

(1994). Usability problem reports: Helping evaluators communicate effectively with developers. In J

Nielsen

Mack

R. L.

(Eds.), Usability inspection methods (pp. 273–294). New York: John Wiley & Sons.

Karat

(1994). A comparison of user interface evaluation methods. In Nielsen

Mack

R. L.

(Eds.), Usability inspection methods (pp. 203–233). New York: John Wiley & Sons.

Keenan

S. L.

(1996). Product usability and process improvement based on usability problem classification. Unpublished doctoral dissertation, Virginia Polytechnic Institute and State University, Blacksburg, VA.

Mack

Montaniz

(1994). Observing, predicting, and analyzing usability problems. In J

Nielsen

Mack

R. L.

(Eds.), Usability inspection methods (pp. 295–339). New York: John Wiley & Sons.

10.

Meister

(1985). Behavioral analysis and measurement methods. New York: John Wiley & Sons.

11.

Muller

M. J.

Dayton

Root

(1993). Comparing studies that compare usability assessment methods: An unsuccessful search for stable criteria. In INTERCHI 1993 Conference Proceedings (Adjunct) (pp. 185–186). New York: ACM Press.

12.

Nielsen

(1993). Usability engineering. Boston: Academic Press.

13.

Nielsen

(1994). Enhancing the explanatory power of usability heuristics. In CHI 1994 Conference Proceedings (pp. 152–158). New York: ACM Press.

14.

Nielsen

Molich

(1990). Heuristic evaluation of user interfaces. In CHI 1990 Conference Proceedings (pp. 249–256). New York: ACM Press.

15.

Norman

D. A.

(1986). Cognitive engineering. In Norman

D. A.

Draper

S. W.

(Eds.), User centered system design: New perspectives on human-computer interaction (pp. 31–61). Hillsdale, NJ: Lawrence Erlbaum Associates.

16.

Rubin

(1994). Handbook of usability testing. New York: John Wiley & Sons.

17.

van Rens

L. S.

(1997). Usability problem classifier. Unpublished master's thesis, Virginia Polytechnic Institute and State University, Blacksburg, VA.

18.

Vora

(1995, May). Classifying user errors in human-computer interactive tasks. Usability Professional's Association Common Ground, 5, 15.