Contemporaneous versus Retrospective User-Reported Critical Incidents in Usability Evaluation

Abstract

The user-reported critical incident technique involves end-users directly in qualitative data collection during formative usability evaluations. An augmented retrospective variation was developed where participants reported incidents while watching a recording of their usability session, rather than reporting incidents contemporaneous to their occurrence during task execution. Retrospective reporting enables controlled comparisons of user-reported and expert-reported methods, since session recordings can be shown to multiple reviewers. It also allows for the collection of incidents without disrupting traditional usability measures, such as time to complete task. A within-subject study with 24 participants found retrospective reporting to be similarly effective to contemporaneous reporting. The study is described and guidelines are provided for the use of both the contemporaneous and augmented retrospective techniques.

Get full access to this article

View all access options for this article.

References

Andre

T. S.

(2000). Determining the effectiveness of the usability problem inspector: a theory-based model and tool for finding usability problems. Unpublished dissertation, Virginia Tech, Blacksburg, VA (available via http://scholar.lib.vt.edu/theses/available/etd-04122000-09440030/).

Bowers

V. A.

Snyder

H. L.

(1990). Concurrent versus retrospective verbal protocol for comparing window usability. In Proceedings of the Human Factors and Ergonomics Society 34th Annual Meeting (pp. 1270–1274). Santa Monica, CA: Human Factors Society.

Capra

M. G.

(2001). An Exploration of End-User Critical Incident Classification. Unpublished thesis, Virginia Tech, Blacksburg, VA (available via http://scholar.lib.vt.edu/theses/available/etd-11122001–143830/).

del Galdo

E. M.

Williges

R. C.

Williges

B. H.

Wixon

D. R.

(1986). An Evaluation of Critical Incidents for Software Documentation Design An Evaluation of Critical Incidents for Software Documentation. In Proceedings of the Human Factors Society 30th Annual Meeting (pp. 19–23). Santa Monica, CA: Human Factors Society.

Ebling

M. R.

John

B. E.

(2000). On the contributions of different empirical data in usability testing. In Proceedings of Designing Interactive Systems 1900: Processes, Practices, Methods, and Techniques (pp. 289–296). New York, NY: ACM Press.

Hartson

H. R.

Castillo

J. C.

(1998). Remote evaluation for post-deployment usability improvement. In Proceedings of the Working Conference on Advanced Visual Interfaces (AVI 1998) (pp. 22–29). New York: ACM Press.

Likert

(1932). A technique for the measurement of attitudes. Archives of Psychology. No., 140, 55.

McCreary

F. A.

(2001). InTouch Usability Evaluation (Technical Report TR-01-11). Blacksburg, VA: Virginia Tech (available via http://eprints.cs.vt.edu:8080/archive/00000535/).

Page

Rahimi

(1995). Concurrent and retrospective verbal protocols in usability testing: is there value added in collecting both? In Proceedings of the Human Factors and Ergonomics Society 39th Annual Meeting (pp. 223–227). Santa Monica, CA: Human Factors and Ergonomics Society.

10.

Thompson

J. A.

(1999). Investigating the Effectiveness of Applying the Critical Incident Technique to Remote Usability Evaluation. Unpublished thesis, Virginia Tech, Blacksburg, VA (available via http://scholar.lib.vt.edu/theses/available/etd-121699–205449/).

11.

Thompson

J. A.

Williges

R. C.

(2000). Web-Based Collection of Critical Incidents During Remote Usability Evaluation. Proceedings of the Human Factors and Ergonomics Society 44th Annual Meeting (pp. 602–605). Santa Monica, CA: Human Factors and Ergonomics Society.