Method Bias and Concurrent Verbal Protocol in Software Usability Testing

Abstract

Concurrent verbal protocols are gaining wide acceptance in software usability testing. In this study, the impact concurrent verbalization has on task performance during a software usability test was investigated. Subjects randomly assigned to two levels of verbalization were asked to complete four tasks of varying difficulty using a disk utility package. Subjects in the verbalization condition were asked to provide an explanation for each step taken to complete a task. Subjects in the control condition were allowed to complete each task silently. Dependent variables were task time, error frequency, and responses to subjective measures of mental workload and ease-of-use. Subjects in the verbalization condition committed fewer errors and consumed less task time than subjects in the silent condition. Further, the mean difference in error frequency and task time between conditions increased with task difficulty. These results were extremely important in revealing a potential method bias in usability tests.

Get full access to this article

View all access options for this article.

References

Bainbridge

(1979). Verbal reports as evidence of process operator's knowledge. International Journal of Man-Machine Studies, 11, 411–436.

Bowers

V. A.

Snyder

H. L.

(1990). Concurrent versus retrospective verbal protocol for comparing window usability. Proceedings of the Human Factors Society 34th Annual Meeting, 1270–1274.

Coleman

W. D.

Williges

R. C.

Wixon

D. R.

(1985). Collecting detailed user evaluations of software interfaces. Proceedings of the Human Factors Society 29th Annual Meeting, 240–244.

Ericsson

K. A.

Simon

H. A.

(1980). Verbal reports as data. Psychological Review, 87, 215–251.

Ericsson

K. A.

Simon

H. A.

(1984) Protocol Analysis. Cambridge: MIT Press.

Gagne

R. H.

Smith

E. C.

(1962). A study of the effects of verbalization on problem solving. Journal of Experimental Psychology, 63, 12–18.

Landauer

T. K.

(1962). Rate of implicit speech. Perceptual and Motor Skills, 15, 646.

Lund

M. L.

(April, 1985). Evaluating the user interface: The candid camera approach. Proceedings of CHI1985. 107–113.

Rhenius

Deffner

(1990). Evaluation of concurrent thinking aloud using eye-tracking data. Proceedings of the Human Factors Society 34th Annual Meeting, 1265–1269.

10.

Schneiderman

(1987). Designing the User Interface: Strategies for Effective Human-Computer Interaction. Reading, MA: Addison-Wesley Publishing Co.

11.

Smead

R. J.

Wilcox

J. B.

Wilkes

R. E.

(1981). How valid are products descriptions and protocols in choice experiments? Journal of Consumer Research, 8, 37–42.

12.

Vidulich

M. A.

Tsang

P. S.

(1985). Assessing subjective workload assessment: A comparison of SWAT and the NASA-Bipolar methods. Proceedings of the Human Factors Society 29th Annual Meeting, 71–75.

13.

Weber

R. J.

Bach

(1969). Visual and speech imagery. British Journal of Psychology, 60, 199–202.