To measure what determines people’s attitudes, definitions, or decisions, surveys increasingly ask respondents to judge vignettes. A vignette typically describes a hypothetical situation or object as having various attributes (dimensions). In factorial surveys, the values (levels) of dimensions are experimentally varied, so that their impact on respondents’ judgments can be estimated. Drawing on the literature in cognitive psychology and survey methodology, we examine two research questions: Does the order in which dimensions are presented impact the vignette evaluations and change substantive conclusions? Under which conditions are order effects mostly likely to occur? Using data from a web survey of 300 students, we analyze several possible moderators: features of the vignette design, characteristics of respondents, and interactions between these features. Results show that strong order effects can occur, but only when the vignettes are of a minimum complexity or respondents show a low attitude certainty.
AbrahamMartinAuspurgKatrinHinzThomas. 2010. “Migration Decisions within Dual-earner Partnerships: A Test of Bargaining Theory.” Journal of Marriage and Family72:876–92.
2.
AiChunrongNortonEdward C.. 2003. “Interaction Terms in Logit and Probit Models.” Economics Letters80:123–29.
3.
AlexanderCheryl S.BeckerHenry Jay. 1978. “The Use of Vignettes in Survey Research.” Public Opinion Quarterly42:93–104.
4.
AlvesWayne M.RossiPeter H.. 1978. “Who Should Get What? Fairness Judgments of the Distribution of Earnings.” American Journal of Sociology84:541–64.
5.
Amaya-AmayaMabelGerardKarenRyanMandy. 2008. “Discrete Choice Experiments in a Nutshell.” Pp. 13-46 in Using Discrete Choice Experiments to Value Health and Health Care, edited byRyanM.GerardK.Amaya-AmayaM.. Dordrecht, the Netherlands: Springer.
6.
AtzmüllerChristianeSteinerPeter M.. 2010. “Experimental Vignette Studies in Survey Research.” Methodology: European Journal of Research Methods for the Behavioral and Social Sciences6:128–38.
AuspurgKatrinHinzThomasLiebigStefan. 2009. “Complexity, Learning Effects, and Plausibility of Vignettes in Factorial Surveys.” Paper presented at the 104th Annual Meeting of the American Sociological Association (ASA), San Francisco, CA.
9.
BennettJeffBlameyRussell. 2001. The Choice Modelling Approach to Environmental Valuation. Cheltenham, UK: Edward Elgar.
10.
BerkRichard A.RossiPeter H.. 1977. Prison Reform and State Elites. Cambridge, MA: Ballinger.
11.
BickartBarbara A.1992. “Question-order Effects and Brand Evaluations: The Moderating Role of Consumer Knowledge.” Pp. 63–79 in Context Effects in Social and Psychological Research, edited bySchwarzN.SudmanS.. New York: Springer.
12.
BijlengaDeniseBonselGouke J.BirnieErwin. 2011. “Eliciting Willingness to Pay in Obstetrics: Comparing a Direct and an Indirect Valuation Method for Complex Health Outcomes.” Health Economics20:1392–406.
13.
BishopGeorgeSmithAndrew. 2001. “Response-order Effects and the Early Gallup Split-ballots.” Public Opinion Quarterly65:479–505.
14.
BorghansLexRomansMargoSauermannJan. 2010. “What Makes a Good Conference? Analysing the Preferences of Labour Economists.” Labour Economics17:868–74.
15.
BradburnNorman M.1992. “What Have We Learned?” Pp. 315–23 in Context Effects in Social and Psychological Research, edited bySchwarzN.SudmanS.. New York: Springer.
16.
BuskensVincentWeesieJeroen. 2000. “An Experiment on the Effects of Embeddedness in Trust Situations.” Rationality and Society12:227–53.
17.
CarrolDouglas J.GreenPaul E.. 1995. “Psychometric Methods in Marketing Research. Part I, Conjoint Analysis.” Journal of Marketing Research32:358–91.
18.
ChristophBernhard. 2005. “Zur Messung des Berufsprestiges. Aktualisierung der Magnitude-Prestigeskala auf die Berufsklassifikation ISCO88.” ZUMA-Nachrichten57:79–127.
19.
ChrzanKeith. 1994. “Three Kinds of Order Effects in Choice-based Conjoint Analysis.” Marketing Letters5:165–72.
20.
CohenJacob. 1969. Statistical Power Analysis for the Behavioral Sciences. New York: Academic Press.
21.
DiefenbachHeikeOppKarl-Dieter. 2007. “When and Why Do People Think There Should Be a Divorce?” Rationality and Society19:485–517.
FarrarShelleyRyanMandy. 1999. “Response-ordering Effects: A Methodological Issue in Conjoint Analysis.” Health Economics8:75–79.
24.
GanongLawrence H.ColemanMarilyn. 2006. “Multiple Segment Factorial Vignette Designs.” Journal of Marriage and Family68:455–68.
25.
GarretKaren. 1982. “Child Abuse: Problems of Definition.” Pp. 177–204 in Measuring Social Judgments. The Factorial Survey Approach, edited byRossiP. H.NockS. L.. Beverly Hills, CA: Sage.
26.
GlenkKlaus. 2006. “Economic Valuation of Biological Diversity. Exploring Non-market Perspectives in the Vicinity of the Lore-Lindu National Park in Indonesia’s Central Sulawesi Region.” Dissertation Thesis, Faculty of Agricultural Sciences, Georg-August-Universität Göttingen, Göttingen, Germany.
27.
GlenkKlaus. 2007. “Effects of Attribute Order in Choice Experiments: Evidence from Rural Indonesia.” Working Paper of the Macaulay Land Use Research Institute, The Macaulay Land Use Research Institute, Aberdeen, Scotland.
28.
HechterMichaelRanger-MooreJamesJassoGuillerminaHorneChristine. 1999. “Do Values Matter? An Analysis of Advance Directives for Medical Treatment.” European Sociological Review15:405–30.
29.
HembroffLarry A.1987. “The Seriousness of Acts and Social Contexts: A Test of Black’s Theory of the Behavior of Law.” American Journal of Sociology93:322–47.
30.
HermkensPiet L. J.BoermanFrank A.. 1989. “Consensus with Respect to the Fairness of Incomes: Differences between Social Groups.” Social Justice Research3:201–15.
31.
HipplerH.-J.SchwarzNorbert. 1986. “Not Forbidding Isn’t Allowing: The Cognitive Basis of the Forbid-allow Asymmetry.” Public Opinion Quarterly50:87–96.
32.
HolbrookAllyson L.GreenMelanie C.KrosnickJon A.. 2003. “Telephone versus Face-to-face Interviewing of National Probability Samples with Long Questionnaires.” Public Opinion Quarterly67:79–125.
33.
HolbrookAllyson L.KrosnickJon A.MooreDavidTourangeauRoger. 2007. “Response Order Effects in Dichotomous Categorical Questions Presented Orally.” Public Opinion Quarterly71:325–48.
34.
HoxJoopKreftItaHermkensPiet. 1991. “The Analysis of Factorial Surveys.” Sociological Methods & Research19:493–510.
35.
JassoGuillermina. 1988. “Whom Shall We Welcome? Elite Judgments of the Criteria for the Selection of Immigrants.” American Sociological Review53:919–32.
36.
JassoGuillermina. 2006a. “Factorial Survey Methods for Studying Beliefs and Judgments.” Sociological Methods & Research34:334–423.
37.
JassoGuillermina. 2006b. “Homans and the Study of Justice.” Pp. 203–27 in George C. Homans: History, Theory, and Method, edited byTrevinoJ.. Boulder, CO: Paradigm Press.
38.
JassoGuillermina. 2012. “Safeguarding Justice Research.” Sociological Methods and Research41:217–39.
39.
JassoGuillerminaOppKarl-Dieter. 1997. “Probing the Character of Norms: A Factorial Survey Analysis of the Norms of Political Action.” American Sociological Review62:947–64.
40.
JassoGuillerminaRossiPeter H.. 1977. “Distributive Justice and Earned Income.” American Sociological Review42:639–51.
41.
JassoGuillerminaWebsterMurrayJr. 1997. “Double Standards in Just Earnings for Male and Female Workers.” Social Psychology Quarterly60:66–78.
42.
JassoGuillerminaWebsterMurrayJr. 1999. “Assessing the Gender Gap in Just Earnings and Its Underlying Mechanisms.” Social Psychology Quarterly62:367–80.
43.
JohnCraig St.BatesNancy A.. 1990. “Racial Composition and Neighborhood Evaluation.” Social Science Research19:47–61.
44.
JohnsonJeff W.2010. “A Heuristic Method for Estimating the Relative Weight of Predictor Variables in Multiple Regression.” Multivariate Behavioral Research35:1–19.
45.
JohnsonRichard M.1989. “Assessing the Validity of Conjoint Analysis.” Pp. 273–280 in Proceeding of the Sawtooth Software Conference “Gaining a Competitive Advantage Through PC-Based Interviewing and Analysis,” edited by MetegranoMargo. Sun Valley, ID: Sawtooth Software.
46.
KjaerTrineBechMickaelGyrd-HansenDorteHart-HansenKristian. 2006. “Ordering Effect and Price Sensitivity in Discrete Choice Experiments: Need We Worry?” Health Economics15:1217–28.
47.
KnäuperBärbelSchwarzNorbertParkDeniseFritschAndreas. 2007. “The Perils of Interpreting Age Differences in Attitude Reports: Question Order Effects Decrease with Age.” Journal of Official Statistics23:515–28.
48.
KrosnickJon A.1988. “Attitude Importance and Attitude Change.” Journal of Experimental Social Psychology24:240–55.
49.
KrosnickJon A.1989. “Attitude Importance and Attitude Accessibility.” Personality and Social Psychology Bulletin15:297–308.
50.
KrosnickJon A.1991. “Response Strategies for Coping with the Cognitive Demands of Attitude Measures in Surveys.” Applied Cognitive Psychology5:213–36.
51.
KrosnickJon A.1992. “The Impact of Cognitive Sophistication and Attitude Importance on Response-order and Question-order Effects.” Pp. 203–18 in Context Effects in Social and Psychological Research, edited bySchwarzN.SudmanS.. New York: Springer.
52.
KrosnickJon A.AlwinDuane F.. 1987. “An Evaluation of a Cognitive Theory of Response-order Effects in Survey Measurement.” Public Opinion Quarterly51:201–19.
53.
KrosnickJon A.PettyRichard E.. 1995. “Attitude Strength: An Overview.” Pp. 1–24 in Attitude Strength: Antecedents and Consequences, edited byPettyR. E.KrosnickJ. A.. Mahwah, NJ: Lawrence Erlbaum.
54.
KuhfeldWarren F.2009. Marketing Research Methods in SAS. Experimental Design, Choice, Conjoint, and Graphical Techniques. Cary, NC: SAS Institute.
55.
KuhfeldWarren F.TobiasRandall D.GarratMark. 1994. “Efficient Experimental Design with Marketing Research Applications.” Journal of Marketing Research31:545–57.
56.
KumarV.GaethGary J.. 1991. “Attribute Order and Product Familiarity Effects in Decision Tasks Using Conjoint Analysis.” International Journal of Research in Marketing8:113–24.
57.
LavineHowardHuffJoseph W.WagnerSteven H.SweeneyDonna. 1998. “The Moderating Influence of Attitude Strength on the Susceptibility to Context Effects in Attitude Surveys.” Journal of Personality and Social Psychology75:359–73.
58.
LiebigStefanMauSteffen. 2002. “Einstellungen zur sozialen Mindestsicherung. Ein Vorschlag zur differenzierten Erfassung normativer Urteile.” Kölner Zeitschrift für Soziologie und Sozialpsychologie54:109–34.
59.
LiebigStefanMauSteffen. 2005. “Wann ist ein Steuersystem gerecht? Einstellungen zu allgemeinen Prinzipien der Besteuerung und zur Gerechtigkeit der eigenen Steuerlast.” Zeitschrift für Soziologie34:468–91.
60.
LouviereJordanHensherDavid A.SwaitJoffre D.. 2000. Stated Choice Methods. Analysis and Application. Cambridge, UK: Cambridge University Press.
61.
LudwickRuthWrightMarion ElizabethZellerRichard AllenDowdingDawn W.LauderWilliamWinchellJanice. 2004. “An Improved Methodology for Advancing Nursing Research: Factorial Surveys.” Advances in Nursing Science27:224–38.
62.
MarkovskyBarryErikssonKimmo. 2012. “Comparing Direct and Indirect Measures of Just Rewards: What Have We Learned?” Sociological Methods and Research41:240–45.
63.
McClendonMcKee J.1991. “Acquiescence and Recency Response-order Effects in Interview Surveys.” Sociological Methods & Research20:60–103.
64.
MeudellBonner M.1982. “Household and Social Standing: Dynamic and Static Dimensions.” Pp. 69–94 in Measuring Social Judgments. The Factorial Survey Approach, edited byRossiP. H.NockS. L.. Beverly Hills, CA: Sage.
65.
MillerJ. L.RossiPeter H.SimpsonJon E.. 1986. “Perceptions of Justice: Race and Gender Differences in Judgments of Appropriate Prison Sentences.” Law & Society Review20:313–34.
66.
Müller-BenedictVolkerTsarouhaElena. 2011. “Können Examensnoten verglichen werden? Eine Analyse von Einflüssen des sozialen Kontextes auf Hochschulprüfungen.” Zeitschrift für Soziologie40:388–409.
67.
NarayanSowmyaKrosnickJon A.. 1996. “Education Moderates Some Response Effects in Attitude Measurement.” Public Opinion Quarterly60:58–88.
68.
NockSteven L.1982. “Family Social Standing: Consensus on Characteristics.” Pp. 95–118 in Measuring Social Judgments. The Factorial Survey Approach, edited byRossiP. H.NockS. L.. Beverly Hills, CA: Sage.
69.
OlsenSøren BøyeLadenburgJacobPetersenMads L.LopdrupUlrichHansenAnja S.DubgaardAlex. 2005. “Motorways versus Nature. A Welfare Economic Valuation of Impacts.” Report No. 5 from the Environmental Assessment Institute Copenhagen, Denmark.
70.
OrmeBryan K.2006. Getting Started with Conjoint Analysis. Strategies for Product Design and Pricing Research. Madison, WI: Research Publishers LLC.
O’TooleRichardWebsterStephen W.O’TooleAnita W.LucalBetsy. 1999. “Teachers’ Recognition and Reporting of Child Abuse: A Factorial Survey.” Child Abuse & Neglect23:1083–101.
73.
PayneStanley L.1949. “Case Study in Question Complexity.” Public Opinion Quarterly13:653–58.
74.
PerreyJesko. 1996. “Erhebungsdesign-Effekte bei der Conjoint-Analyse.” Marketing. Zeitschrift für Forschung und Praxis2:105–16.
75.
RogersWilliam H.1993. “Regression Standard Errors in Clustered Samples.” Stata Technical Bulletin3:19–23.
76.
RossiPeter H.1979. “Vignette Analysis: Uncovering the Normative Structure of Complex Judgments.” Pp. 176–86 in Qualitative and Quantitative Social Research. Papers in Honor of Paul F. Lazarsfeld, edited byMertonR. K.ColemanJ. S.RossiP. H.. New York: The Free Press.
77.
RossiPeter H.AndersonAndy B.. 1982. “The Factorial Survey Approach: An Introduction.” Pp. 15–67 in Measuring Social Judgements: The Factorial Survey Approach, edited byRossiP. H.NockS. L.. Beverly Hills, CA: Sage.
78.
RossiPeter H.SampsonWilliam A.BoseChristine E.JassoGuillerminaPasselJeff. 1974. “Measuring Household Social Standing.” Social Science Research3:169–90.
79.
RyanMandyGerardKarenAmaya-AmayaMabel. 2008. Discrete Choice Experiments to Value Health and Health Care. Dordrecht, the Netherlands: Springer.
80.
SauerCarstenAuspurgKatrinHinzThomasLiebigStefan. 2011. “The Application of Factorial Surveys in General Populations Samples: The Effects of Respondent Age and Education on Response Times and Response Consistency.” Survey Research Methods5:89–102.
81.
SauerCarstenAuspurgKatrinHinzThomasLiebigStefanSchuppJürgen. 2014. “Method Effects in Factorial Surveys: An Analysis of Respondents’ Comments, Interviewers’ Assessments, and Response Behavior.” SOEPpaper on Multidisciplinary Panel Data Research No. 629. Berlin, Germany: German Institute for Economic Research (DIW). Retrieved June 6, 2015 (http://www.diw.de/documents/publikationen/73/diw_01.c.438559.de/diw_sp0629.pdf).
82.
SchumanHoward. 1992. “Context Effects: State of the Past/State of the Art.” Pp. 5–20 in Context Effects in Social and Psychological Research, edited bySchwarzN.SudmanS.. New York: Springer.
83.
SchumanHowardPresserStanley. 1981. Questions and Answers in Attitude Surveys. Experiments on Question Form, Wording, and Context. New York: Academic Press.
84.
SchwarzNorbert. 2007. “Cognitive Aspects of Survey Methodology.” Applied Cognitive Psychology21:277–87.
85.
SchwarzNorbertHipplerH.-J.Noelle-NeumannElisabeth. 1992. “A Cognitive Model of Response-order Effects in Survey Measurement.” Pp. 187–201 in Context Effects in Social and Psychological Research, edited bySchwarzN.SudmanS.. New York: Springer.
86.
SchwarzNorbertKnäuperBärbel. 2000. “Cognition, Aging and Self-reports.” Pp. 233–52 in Cognitive Aging: A Primer, edited byParkD. C.SchwarzN.. Philadelphia, PA: Psychology.
87.
ScottAnthonyVickSandra. 1999. “Patients, Doctors and Contracts: An Application of Principal-agent Theory to the Doctor-patient Relationship.” Scottish Journal of Political Economy46:111–34.
88.
SharonRuth GrossHoltzRolfMillerNorman. 1995. “Attitude Certainty.” Pp. 215–46 in Attitude Strength: Antecedents and Consequences, edited byPettyR. E.KrosnickJ. A.. Mahwah, NJ: Lawrence Erlbaum.
89.
ShepelakNorma J.AlwinDuane F.. 1986. “Beliefs about Inequality and Perceptions of Distributive Justice.” American Sociological Review51:30–46.
90.
SimonHerbert. 1957. Models of Man. New York: Wiley.
91.
SmithTom W.1992. “Thoughts on the Nature of Context Effects.” Pp. 163–84 in Context Effects in Social and Psychological Research, edited bySchwarzN.SudmanS.. New York: Springer.
92.
SoofiEhsan S.RetzerJoseph J.Yasai-ArdekaniMasoud. 2000. “A Framework for Measuring the Importance of Variables with Applications to Management Research and Decision Models.” Decision Sciences31:595–625.
93.
SoperJohn C.WalstadWiliam B.. 1987. Test of Economic Literacy. Examiner’s Manual. New York: Joint Council on Economic Education.
94.
StarkGunnarLiebigStefanWegenerBernd. 2008. “Gerechtigkeitsideologien.” In Zusammenstellung sozialwissenschaftlicher Items und Skalen. Online-Handbook of Social Measurement Instruments, edited by Glockner-RistA.. Bonn, Germany: GESIS-Onlinehandbook. Retrieved June 7, 2015 (http://zis.gesis.org/ZisApplication/).
SteinerPeterAtzmüllerChristiane. 2006. “Experimentelle Vignettendesigns in Faktoriellen Surveys.” Kölner Zeitschrift für Soziologie und Sozialpsychologie58:117–46.
97.
SudmanSeymourBradburnNorman M.SchwarzNorbert. 1996. Thinking about Answers: The Application of Cognitive Processes to Survey Methodology. New York: Jossey-Bass.
98.
TormalaZakary L.RuckerDerek D.. 2007. “Attitude Certainty: A Review of Past Findings and Emerging Perspectives.” Social and Personality Psychology Compass1:469–92.
99.
TourangeauRoger. 1999. “Context Effects on Answers to Attitude Questions.” Pp. 111–32 in Cognition and Survey Research, edited bySirkenM. G.HerrmannD. J.SchechterS.SchwarzN.TanurJ. M.TourangeauR.. New York: John Wiley.
TourangeauRogerRasinskiKenneth A.BradburnNormanD’AndradeRoy. 1989a. “Belief Accessibility and Context Effects in Attitude Measurement.” Journal of Experimental Social Psychology25:401–21.
102.
TourangeauRogerRasinskiKenneth A.BradburnNormanD’AndradeRoy. 1989b. “Carryover Effects in Attitude Surveys.” The Public Opinion Quarterly53:495–524.
103.
TourangeauRogerRipsLance J.RasinskiKenneth A.. 2000. The Psychology of Survey Response. New York: Cambridge University Press.
104.
TourangeauRogerSingerEleanorPresserStanley. 2003. “Context Effects in Attitude Surveys: Effects on Remote Items and Impact on Predictive Validity.” Sociological Methods & Research31:486–513.
105.
WallanderLisa. 2009. “25 Years of Factorial Surveys in Sociology: A Review.” Social Science Research38:505–20.
106.
WooldridgeJeffey M.2003. Introductory Econometrics. A Modern Approach. Mason, OH: South Western.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
