Recent research has shown that even minor changes in the visual layout of survey questions can affect the way in which respondents answer. Still, little is known about how these changes affect the actual answering process. The recent development of client-side paradata allows us to better understand the way respondents construct their answers by providing new types of data, including time taken to answer a question and whether respondents change their answers. Although a number of studies have used this technology to examine response latency, few have taken advantage of its potential to collect data on the types of changes respondents make while answering survey questions. This study uses client-side paradata to analyze changes in multiple versions of three types of Web survey questions. The results suggest that the visual layout of survey questions not only affects the number but also the types of changes respondents make.
Bassili, J.N.1996. The how and the why of response latency in telephone surveys . In Answering questions, ed. N. Schwarz and S. Sudman, 319-46. San Francisco: Jossey-Bass.
2.
Bassili, J.N., and J.F. Fletcher.1991. Response time measurement in survey research. A method for CATI and a new look at nonattitudes. Public Opinion Quarterly55 (3): 331-46.
3.
Bassili, J.N., and B.S. Scott.1996. Response latency and question problems . Public Opinion Quarterly60 (3): 390-99.
4.
Blair, J., F. Conrad, A.C. Ackermann, and G. Claxton.2006. The effect of sample size on cognitive interview findings. Paper presented at the 61st annual meeting of the American Association for Public Opinion Research, Montreal, Quebec, May 18-21.
5.
Bosnjak, M.T., and W. Bandilla.2001. Participation in Web surveys. A typology . Zuma-Nachrichten25 (48): 7-17.
6.
Christian, L.M.2003. The influence of visual layout on scalar questions in Web surveys. Unpublished master's thesis, Washington State University.
7.
Christian, L.M., and D.A. Dillman.2004. The influence of graphical and symbolic language manipulations on responses to self-administered questions. Public Opinion Quarterly68 (1): 58-81.
8.
Christian, L.M., N.L. Parsons, and D.A. Dillman.2005. Understanding the effects of visual design and layout. Working paper, Social and Economic Sciences Research Center , Washington State University, Pullman.
9.
Converse, P.E.1964. New dimensions of meaning for cross-section sample surveys in politics. International Social Science Journal16:19-34.
10.
Couper, M.P., R. Tourangeau, and F.G. Conrad.2004. What they see is what we get: Response options for Web surveys. Social Science Computer Review22 (1): 111-27.
11.
Dillman, D.A.2007. Mail and Internet surveys: The tailored design method 2007 update. New York: John Wiley .
12.
Dillman, D.A., and L.M. Christian.2005. Survey mode as a source of instability in responses across surveys. Field Methods17 (1): 30-52.
13.
Dillman, D.A., and C. Redline.2004. Concepts and procedures for testing paper self-administered questionnaires: Cognitive interviews and field test questions. In Methods for testing and evaluating survey questionnaires , ed. S. Presser, J. Rothberg, M. Couper, J. Lessler, E. Martin, J. Martin, and E. Singer , 297-319. New York: John Wiley.
14.
Dillman, D.A., J.D. Smyth, L.M. Christian, and M.J. Stern.2003. Multiple-answer questions in self-administered surveys: The use of check-all-that-apply and forced-choice question formats . Paper presented at the 165th annual meeting of the American Statistical Association, San Francisco, August 3-7.
15.
Forsyth, B.H., and J.T. Lessler.1991. Cognitive laboratory methods: A taxonomy . In Measurement errors in surveys, ed. P. Biemer, R. Groves, L. Lysberg, N. Mathiowetz , and S. Sudman, 393-418. New York: John Wiley.
16.
Healey, B.2007. Drop downs and scroll mice: The effect of response option format and input mechanism employed on data quality in Web surveys. Social Science Computer Review25 (1): 111-28.
17.
Heerwegh, D.2002. Describing response behavior in Web surveys using client-side paradata. Paper presented at the International Workshop on Web Surveys, Mannheim, Germany, October 17-19.
18.
---. 2003. Explaining response latency and changing answers using client-side paradata from a Web survey. Social Science Computer Review21 (3): 360-73.
19.
Israel, G.D., and C.L. Taylor.1990. Can response order bias evaluations ? Evaluation and Program Planning13 (4): 365-71.
20.
Jeavons, A.1999. Ethology and the Web. Observing respondent behavior in Web surveys. Marketing Research Today28 (2): 69-76.
21.
Krosnick, J.A.1991. Response strategies for coping with the cognitive demands of attitude measure in surveys. Applied Cognitive Psychology5 (3): 213-36.
22.
Krosnick, J.A., and D.F. Alwin.1987. An evaluation of a cognitive theory of response-order effects in survey measurement. Public Opinion Quarterly51 (2): 201-19.
23.
Poe, G.S., I. Seeman, J. McLaughlin, E. Mehl, and M. Dietz.1988. "Don't know" boxes in factual questions in a mail questionnaire. Public Opinion Quarterly52 (2): 212-22.
24.
Presser, S., and J. Blair.1994. Survey pretesting: Do different methods produce different results? Sociological Methodology24:73-104.
25.
Presser, S., M. Couper, J. Lessler, E. Martin, J. Martin, J. Rothberg, and E. Singer.2004. Methods for testing and evaluating survey questions. Public Opinion Quarterly68 (1): 109-30.
26.
Schwarz, N.1996. Cognition and communication: Judgmental biases, research methods, and the logic of conversation. Mahwah, NJ: Lawrence Erlbaum.
27.
Smith, T.1993. Little things matter: A sampler of how differences in questionnaire format can affect survey response. National Opinion Research Center, Methodological Report no. 78. University of Chicago General Society Survey.
28.
Smyth, J.D., D.A. Dillman, L.M. Christian, and M.J. Stern.2006. Comparing check-all and forced-choice question formats in Web surveys. Public Opinion Quarterly70 (1): 66-77.
29.
Stern, M.J., D.A. Dillman, and J.D. Smyth.2007. Visual design, order effects and respondent characteristics in a self-administered survey. Survey Research Methods1 (3): 121-38.
30.
Stern, M.J., D.A. Dillman, J.D. Smyth, and L.M. Christian.2004. The use of paradata for evaluating alternate versions of Web survey questions. Paper presented at the 57th annual meeting of the World Association of Public Opinion Research , Phoenix, Arizona, May 11-13.
31.
Sudman, S., N. Bradburn, and N. Schwarz.1996. Thinking about answers: The application of cognitive processes to survey methodology. San Francisco : Jossey-Bass.
32.
Thomas, R., and J. Klein.2006. Merely incidental?: Effects of response format on self-reported behavior. Journal of Official Statistics22 (2): 221-44.
33.
Tourangeau, R., K.A. Rasinski, and N. Bradburn.1991. Measuring happiness in surveys: A test of the subtraction hypothesis. Public Opinion Quarterly55 (2): 255-66.
