Different Order,Different Results? The Effects of Dimension Order in Factorial Survey Experiments

Abstract

Factorial surveys are widely used in the social sciences to measure respondents’ attitudes, beliefs, or behavioral intentions. In such surveys, respondents evaluate short descriptions of hypothetical situations, persons, or objects that vary across several dimensions. An important prerequisite of the method’s validity is that respondents are able to deal with the highly complex task created by the need to consider several variable dimensions within one coherent judgment. We analyze the effects of the order in which dimensions are presented in running text vignettes. An experimental setup with four order treatments was randomly allocated to 787 respondents (based on a random sample of register data), yielding 3,119 vignette evaluations. The analyses compare respondent groups across age, education, and response speed. Overall, there is no strong evidence for order effects. However, we find a slight tendency for fast responders to be more prone to recency effects.

Get full access to this article

View all access options for this article.

References

Auspurg

Hinz

. 2015. Factorial survey experiments. Thousand Oaks, CA: Sage.

Auspurg

Jäckle

. 2015. First equals most important? Order effects in vignette-based measurement. Sociological Methods & Research 46:490–539.

Ayidiya

S. A.

McClendon

M. J.

. 1990. Response effects in mail surveys. Public Opinion Quarterly 54:229–47.

Beniger

J. R.

Gusek

J. A.

. 1995. The cognitive revolution in public opinion and communication research. In Public opinion and the communication of consent, edited by Glasser

T. L.

Salmon

C. T.

, 217–48. New York: Guilford.

Ceci

S. J.

1991. How much does schooling influence general intelligence and its cognitive components? A reassessment of evidence. Developmental Psychology 27:703–22.

Hinz

Joos

Schleicher

. 2012. Bildung und Lebenszufriedenheit. Ergebnisse der Konstanzer Bürgerbefragung 2011–4. Welle. Statistik Bericht 5. Konstanz, Germany: University of Konstanz.

Holbrook

A. L.

Krosnick

J. A.

Moore

Tourangeau

. 2007. Response order effects in dichotomous categorical questions presented orally. Public Opinion Quarterly 71:325–48.

Knäuper

1999. The impact of age and education on response order effects in attitude measurement. Public Opinion Quarterly 63:347–70.

Krosnick

J. A.

1989. Attitude importance and attitude accessibility. Personality and Social Psychology Bulletin 15:297–308.

10.

Krosnick

J. A.

1991. Response strategies for coping with the cognitive demands of attitude measures in surveys. Applied Cognitive Psychology 5:213–36.

11.

Krosnick

J. A.

Alwin

D. F.

. 1987. An evaluation of a cognitive theory of response-order effects in survey measurement. Public Opinion Quarterly 51:201–19.

12.

Kuhfeld

W. F.

Tobias

R. D.

Garratt

. 1994. Efficient experimental design with marketing research applications. Journal of Marketing Research 31:545–57.

13.

Mayerl

Sellke

Urban

. 2005. Analyzing cognitive processes in CATI-surveys with response latencies: An empirical evaluation of the consequences of using different baseline speed measures. Schriftenreihe des Instituts für Sozialwissenschaften der Universität Stuttgart 2:1–18.

14.

McClendon

M. J.

1991. Acquiescence and recency response-order effects in interview surveys. Sociological Methods & Research 20:60–103.

15.

Narayan

Krosnick

J. A.

. 1996. Education moderates some response effects in attitude measurement. Public Opinion Quarterly 60:58–88.

16.

Robbins

Kiser

. 2018. Legitimate authorities and rational taxpayers: An investigation of voluntary compliance and method effects in a survey experiment of income tax evasion. Rationality and Society 30:247–301.

17.

Rogers

W. H.

1993. Regression standard errors in clustered samples. Stata Technical Bulletin 3:19–23.

18.

Sauer

Auspurg

Hinz

Liebig

. 2011. The application of factorial surveys in general population samples: The effects of respondent age and education on response times and response consistency. Survey Research Methods 5:89–102.

19.

Schaie

K. W.

1996. Intellectual development in adulthood. The Seattle Longitudinal Study. Cambridge: Cambridge University Press.

20.

Schwarz

2007. Cognitive aspects of survey methodology. Applied Cognitive Psychology 21:277–87.

21.

Schwarz

Knäuper

. 2000. Cognition, aging, and self-reports. In Cognitive aging: A primer, edited by Park

Schwarz

, 233–52. Philadelphia: Psychology Press.

22.

Schwarz

Strack

Hippler

H.-J.

Bishop

. 1991. The impact of administration mode on response effects in survey measurement. Applied Cognitive Psychology 5:193–212.

23.

Urban

Mayerl

. 2007. Antwortlatenzzeiten in der survey-basierten Verhaltensforschung. Kölner Zeitschrift für Soziologie und Sozialpsychologie 59:692–713.

24.

Wallander

. 2009. 25 years of factorial surveys in sociology: A review. Social Science Research 38:505–20.

25.

Wooldridge

J. M.

2009. Introductory econometrics. A modern approach, 4th ed. Mason, OH: South Western.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

1.35 MB