Are We Overlooking Some Usability Testing Methods? A Comparison of Lab,Beta,and Forum Tests

Abstract

We compared the effectiveness of Lab testing, Beta testing, and Forum testing at identifying software usability problems. Thirty participants were involved in the experiment, with ten participants in each of the three test conditions. The Lab test involved participants performing prescribed scenarios with the software in a controlled Lab environment, while human factors engineers recorded participant's problems. The Beta test method had participants use the software in their own environment to perform their real world work, and record their own problems. The Forum test was similar to the Beta test, except that the software was made available on a company-wide computer bulletin board and the participants selected themselves. Findings show that the Beta test method was as effective as the Lab test method in the number of problem types identified. The Lab test uncovered a larger proportion of serious usability problems than did the Beta test. The Beta test method was the most cost-effective method. The Forum test method found the fewest number of problem types, and was the least cost-effective. Thus, the results of this study broaden the current literature by showing that the Beta test method may be a cost-effective alternative to the traditional lab test.

Get full access to this article

View all access options for this article.

References

Desurvire

Lawrence

Atwood

(1991). Empiricism versus judgement: Comparing user interface evaluation methods on a new telephone-based interface. ACM SIGCHI Bulletin, 23, 58–59.

Edwards

A. L.

(1960). Experimental design in psychological research. New York: Holt, Rinehart and Winston.

Jeffries

Miller

J.R.

Wharton

Uyeda

K.M.

(1991). User interface evaluation in the real world: A comparison of four techniques. In CHI′91 Conference Proceedings (pp. 119–124). Reading, MA: Addison Wesley.

Karat

C.-M.

Campbell

Fiegel

(1992). Comparisons of empirical testing and walkthrough methods in user interface evaluation. In CHI′92 Conference Proceedings (pp. 397–404). Reading, MA: Addison Wesley.

Lamb

D. A.

(1988). Software engineering: Planning for change. Englewood Cliffs, NJ: Prentice Hall.

Lewis

Polson

Wharton

Rieman

(1990). Testing a walkthrough methodology for theory-based design of walk-up-and-use interfaces. In CHI′90 Conference Proceedings (pp. 235–242). Reading, MA: Addison Wesley.

Molich

Nielsen

(1990). Improving a human-computer dialogue. Communications of the ACM, 33, 338–348.

Nielsen

(1992). The usability engineering life cycle. Computer, 25, 12–22.

Nielsen

Molich

(1990). Heuristic evaluation of user interfaces. In CHI′90 Conference Proceedings (pp. 249–256). Reading, MA: Addison Wesley.

10.

Virzi

R. A.

(1992). Refining the test phase of usability evaluation: How many subjects is enough? Human Factors, 34, 457–468.

11.

Wharton

Bradford

Jeffries

Franzke

(1992). Applying cognitive walkthroughs to more complex interfaces: Experiences, issues, and recommendations. In CHI′92 Conference Proceedings (pp. 381–388). Reading, MA: Addison Wesley.

12.

Wyatt

A. L.

(1990). Computer professional's dictionary. Berkeley, CA: Osborne McGraw-Hill.