Comparing Usability Problem Identification and Description by Practitioners and Students

Abstract

Many studies of usability studies count the number of usability problems identified to measure the effectiveness of an evaluation. However, communicating problems is also important to evaluation effectiveness, because a problem found but poorly explained may not be fixed. This study compared lists of usability problems from 21 practitioners and 23 students watching a pre-recorded usability session. Lists were evaluated for the number of problems reported, and for following six guidelines for describing usability problems: be clear and avoid jargon, describe problem severity, provide backing data, describe problem causes, describe user actions, and provide a solution. There was no difference in the number of problems reported by students and practitioners, but there was a difference in their ratings for following several of the guidelines. Using both measures provides a more complete assessment of usability reports.

Get full access to this article

View all access options for this article.

References

Capra (2005). Factor Analysis of Card Sort Data: An Alternative to Hierarchical Cluster Analysis. In Proceedings of the Human Factors and Ergonomics Society 49th Annual Meeting (pp. 691–696). Santa Monica, CA: HFES.

Capra (2006). Usability Problem Description and the Evaluator Effect in Usability Testing. Unpublished dissertation, Virginia Tech: Blacksburg. Available via http://scholar.lib.vt.edu/theses/available/etd-03222006-201913/.

Capra Smith-Jackson (2006). Developing Guideines for Describing Usability Problems. Technical Report #ACE/HCI-2005–002. Blacksburg: Virginia Tech, Grado Department of Industrial and Systems Engineering. Available via http://ace.ise.vt.edu/publicationstechnicalreports.htm

Dumas

J. S.

Molich

Jeffries

(2004). Describing usability problems: Are we sending the right message? interactions, 11(4), 24–29.

Hertzum

Jacobsen

N. E.

(2003). The evaluator effect: A chilling fact about usability evaluation methods. International Journal of Human-Computer Interaction, 15(1), 183–204.

Jacobsen

N. E.

Hertzum

John

B. E.

(1998). The Evaluator Effect in Usability Tests. In Karat

C.-M.

Lund

Coutaz

Karat

, Proceedings of the Conference on Human Factors in Computing Systems (CHI 98) (pp. 255–256). New York: ACM.

Long

Styles

Andre

Malcolm

(2005). Usefulness of Nonverbal Cues from Participants in Usability Testing Sessions. In Salvendy

Proceedings of the 11th International Conference on Human-Computer Interaction (HCII 2005). St. Louis, MO: Mira Digital.

Molich

(2004). Comparative Usability Evaluation - CUE. Dialog Design: Stenløse, Denmark. Retrieved August 8, 2004 from http://www.dialogdesign.dk/cue.html.

Molich

Dumas

J. S.

(in press). Comparative Usability Evaluation 4 (CUE-4). Behaviour & Information Technology.

10.

Molich

Ede

M. R.

Kaasgaard

Karyukin

(2004). Comparative Usability Evaluation. Behaviour & Information Technology, 23(1), 65–74

11.

Molich

Thomsen

A. D.

Karyukina

Schmidt

Ede

Van Oel

, et al. (1999). Comparative evaluation of usability tests. In Proceedings of the Conference on Human Factors in Computing Systems (CHI '98) [extended abstracts]. New York: ACM.

12.

Rourke

(2003). CUE-4: Lessons in Best Practice for Usability Testing and Expert Evaluation. User Vision. Retrieved December 14, 2004 from http://www.uservision.co.uk/usability_articles/usability_CUE.asp.

13.

Skov

M. B.

Stage

(2005). Supporting problem identification in usability evaluations. In Proceedings of the 19th conference of the computer-human interaction special interest group (CHISIG) of Australia on Computer-human interaction: citizens online: considerations for today and the future (OZCHI 2005). Narrabundah, Australia: Computer-Human Interaction Special Interest Group (CHISIG) of Australia.

14.

Vermeeren

A. P. O. S.

Van Kesteren

I. E. H.

Bekker

M. M.

(2003). Managing the Evaluator Effect in User Testing. In Rauterberg

Proceedings of the Human-Computer Interaction - INTERACT'03 (pp. 647–654): IOS.