Effect of Level of Problem Description on Problem Discovery Rates: Two Case Studies

Abstract

The primary purpose of this analysis was to investigate the effect of changing the level of description of usability problems on the estimate of the problem discovery rate (p). A secondary purpose was to describe a method for using p to estimate the number of problems remaining available for discovery given the constraints associated with a particular participant population, application, and set of tasks. The level of problem description influenced estimates of p, and the direction of influence was predictable, with higher levels of description producing higher estimates of p. Practitioners need a level of description that flows easily into recommendations for redesigning products, but to keep usability studies as efficient as possible, practitioners also need to seek a level of description that takes advantage of common patterns in observed usability problems. Managing this tradeoff is only one of the challenges of usability evaluation, but it is an important one.

Get full access to this article

View all access options for this article.

References

Cockton

Lavery

(1999). A framework for usability problem extraction. In Human-Computer Interaction — INTERACT 99 (pp. 344–352. Amsterdam: IOS Press.

Connell

I. W.

Hammond

N. V.

(1999). Comparing usability evaluation principles with heuristics: Problem instances vs. problem types. Human-Computer Interaction — INTERACT 99 (pp. 621–629. Amsterdam: IOS Press.

Gillan

D. J.

Bias

R. G.

Usability science I: Foundations. International Journal of Human-Computer Interaction, 13, 351–372.

Hertzum

Jacobsen

N. J.

(2003). The evaluator effect: A chilling fact about usability evaluation methods. International Journal of Human-Computer Interaction, 15, 183–204.

Lavery

Cockton

Atkinson

M. P.

(1997). Comparison of evaluation methods using structured usability problem reports. Behaviour & Information Technology, 16, 246–266.

Lee

W. O.

(1998). Analysis of problems found in user testing using an approximate model of user action. In People and Computers XIII: Proceedings of HCI 98 (pp. 23–35. Sheffield, UK: Springer-Verlag.

Lewis

J. R.

(1994). Sample sizes for usability studies: Additional considerations. Human Factors, 36, 368–378.

Lewis

J. R.

(2001). Evaluation of procedures for adjusting problem-discovery rates estimated from small samples. International Journal of Human-Computer Interaction, 13, 445–480.

Lewis

J. R.

(2006). Usability testing. In Salvendy

Handbook of Human Factors and Ergonomics (pp. 1275–1316. New York, NY: John Wiley.

10.

Limin

Salvendy

Turley

(2002). Effectiveness of user testing and heuristic evaluation as a function of performance classification. Behaviour & Information Technology, 21, 137–143.

11.

Nielsen

(1994). Heuristic evaluation. In Nielsen

Mack

R. L.

Usability Inspection Methods (pp. 25–61. New York, NY: John Wiley.

12.

Nielsen

Landauer

T. K.

(1993). A mathematical model of the finding of usability problems. In Conference Proceedings on Human Factors in Computing Systems — CHI93 (pp. 206–213. New York, NY: Association for Computing Machinery.

13.

Nielsen

Molich

(1990). Heuristic evaluation of user interfaces. In Conference Proceedings on Human Factors in Computing Systems — CHI90 (pp. 249–256. New York, NY: ACM.

14.

Virzi

R. A.

(1990). Streamlining the design process: Running fewer subjects. In Proceedings of the Human Factors Society 34th Annual Meeting (pp. 291–294. Santa Monica, CA: Human Factors Society.

15.

Virzi

R. A.

(1992). Refining the test phase of usability evaluation: How many subjects is enough? Human Factors, 34, 443–451.