Qualitative Data Differences between a Focus Group and Online Forum Hosting a Usability Design Review

Abstract

Traditional usability assessment test methods often do not accommodate conventional resource constraints which typically result in testing renouncement; thus, new methods are needed. This case study reviews a new test method, the internet forum, and compares data outcomes related to participant participation between this forum and the traditional focus group. The same design review was hosted in all groups: two forum groups (n=28 and 7) and two focus group sessions (n=14 and 10). Data was coded to determine what percentage of the users responded to each posed question, how many user conversations occurred and how many design suggestions were provided. Thus, more participants responded to questions in the forum rather than the focus group but more forum conversations occurred comparatively. However, these conversations did not inspire more designs suggestions comparatively. Also, forum participants most often directly answered the questions and provided more design suggestions directly related to the questions posed while focus group participants offered more off-topic responses that provided important rich contextual information (e.g., explanation behind user requirements). Research implications, threats to validity and future work are also discussed.

Get full access to this article

View all access options for this article.

References

Faulkner

(2003). Beyond the five-user assumptions: Benefits of increased sample sizes in usability testing. Behavior Research Methods, Instruments and Computers, 35(3), 379-383.

Fern

E. F.

(1982). The use of focus groups for idea generation: The effects of group size, acquaintanceship, and moderator on response quantity and quality. Journal of Marketing Research, 19(1), 1-13.

Fitts

P. M.

(1951). Engineering psychology, and equipment design. In Stevens

S. S.

(Ed.), Handbook of Experimental Psychology (pp. 1287-1340). New York: Wiley.

Glaser

B. G.

Strauss

A. L.

(1967). The discovery of grounded theory. Chicago: Aldine.

Joinson

(1998). Causes and implications of disinhibited behavior on the Internet. In Gackenbach

(ed.); Psychology and the Internet: Intrapersonal, Interpersonal, and Transpersonal Implications. Academic Press, San Diego, CA, pp. 43-60.

Kitzinger

(1994). The methodology of focus groups: The importance of interaction between research participants. Sociology of Health & Illness, 16 (1), 103-121.

Nielsen

(2000). Why you only need to test with 5 users, Alertbox, http://www.useit.com/alertbox/20000319.html

Rice

R. E.

Love

(1987). Electronic emotion: a content and network analysis of a computer-mediated communication network, Communication Research 14, 85-105.

Spool

Schroeder

(2001). Testing websites: five users is nowhere near enough. In CHI 2001 Extended Abstracts, ACM, 285-286.

10.

Steward

D. W.

Shambdasani

P. N.

Rook

D. W.

(2007). Focus groups: Theory and practice (2nd ed.). Thousand Oaks, CA: Sage.

11.

Suler

(2004). The online disinhibition effect. CyberPsychology & Behavior, 7(3), 321-326.

12.

Virzi

R. A.

(1992). Refining the test phase of usability evaluation: How many subjects is enough? Human Factors, 34, 457-468.

13.

Ward

V. M.

Bertrand

J. T.

Brown

L. F.

(1991). The comparability of focus group and survey results. Evaluation Review, 15(2), 266-283.

14.

Yin

R. K.

(2003). Case Study Research: Design and Methods (3rd ed.). Thousand Oaks, CA: Sage Publications.