In factorial survey designs, respondents evaluate multiple short descriptions of social objects (vignettes) that experimentally vary different levels of attributes of interest. Analytical methods (including individual-level regression analysis and multilevel models) estimate the weights (or utilities) assigned to the levels of the different attributes by participants to arrive at an overall response to the vignettes. In the current paper, we explain how data from factorial surveys can be analyzed in a structural equation modeling framework using an approach called structural equation modeling for within-subject experiments. We review the use of factorial surveys in social science research, discuss typically used methods to analyze factorial survey data, introduce the structural equation modeling for within-subject experiments approach, and present an empirical illustration of the proposed method. We conclude by describing several extensions, providing some practical recommendations, and discussing potential limitations.
Aiman-SmithLyndaScullenSteven E.BarrSteve H.. 2002. “Conducting Studies of Decision Making in Organizational Contexts: A Tutorial for Policy-Capturing and Other Regression-Based Techniques.” Organizational Research Methods5(4):388–414.
2.
AuspurgKatrinHinzThomas. 2015. Factorial Survey Experiments. Quantitative Applications in the Social Sciences series, Vol. 175. Los Angeles: Sage Publications.
3.
AuspurgKatrinJäckleAnnette. 2017. “First Equals Most Important? Order Effects in Vignette-Based Measurement.” Sociological Methods & Research46(3):490–539.
4.
BeierleinConstanzeKuntzAnabelDavidovEldad. 2016. “Universalism, Conservation and Attitudes Toward Minority Groups.” Social Science Research58:68–79.
5.
BillietJaak B.DavidovEldad. 2008. “Testing the Stability of an Acquiescence Style Factor Behind Two Interrelated Substantive Variables in A Panel Design.” Sociological Methods and Research36(4):542–62.
6.
BillietJaak B.McClendonMcKee J.. 2000. “Modeling Acquiescence in Measurement Models for Two Balanced Sets of Items.” Structural Equation Modeling7(4):608–28.
7.
CarrollJ. DouglasGreenPaul E.. 1995. “Psychometric Methods in Marketing Research: Part I, Conjoint Analysis.” Journal of Marketing Research32(4):385–91.
8.
CastilloJuan C.OlivosFranciscoAzarAriel. 2019. “Deserving A Just Pension: A Factorial Survey Approach.” Social Science Quarterly100(1):359–78.
9.
CieciuchJanDavidovEldadSchmidtPeterAlgesheimerRenéSchwartzShalom H.. 2014. “Comparing Results of an Exact vs. an Approximate (Bayesian) Measurement Invariance Test: A Cross-Country Illustration with A Scale to Measure 19 Human Values.” Frontiers in Psychology5:982.
10.
ColeDavid A.PerkinsCorinne E.ZelkowitzRachel L.. 2016. “Impact of Homogeneous and Heterogeneous Parceling Strategies When Latent Variables Represent Multidimensional Constructs.” Psychological Methods21(2):164–74.
11.
DavidovEldadMeulemanBart. 2012. “Explaining Attitudes Towards Immigration Policies in European Countries: The Role of Human Values.” Journal of Ethnic and Migration Studies38(5):757–75.
12.
DavidovEldadSchmidtPeterSchwartzShalom H.. 2008. “Bringing Values Back In: The Adequacy of the European Social Survey to Measure Values in 20 Countries.” Public Opinion Quarterly72(3):420–45.
13.
Di StasioValentinaGerxhaniKlarita. 2015. “Employers’ Social Contacts and Their Hiring Behavior in A Factorial Survey.” Social Science Research51:93–107.
14.
DohertyMichael E.KeeleyStuart M.. 1972. “Use of Subjective Predictors in Regression Analysis for Policy Capturing.” Journal of Applied Psychology56(3):277–78.
15.
DülmerHermann. 2007. “Experimental Plans in Factorial Surveys: Random or Quota Design?” Sociological Methods & Research35(3):382–409.
16.
DülmerHermann. 2016. “The Factorial Survey: Design Selection and its Impact on Reliability and Internal Validity.” Sociological Methods & Research45(2):304–47.
17.
EmersonMichael O.ChaiKaren J.YanceyGeorge. 2001. “Does Race Matter in Residential Segregation? Exploring the Preferences of White Americans.” American Sociological Review66(6):922–35.
18.
ESS, European Social Survey. 2001. European Social Survey Core Questionnaire Development - Overview. London: Centre for Comparative Social Surveys, City University London.
19.
FingerClaudia. 2016. “Institutional constraints and the Translation of College Aspirations Into Intentions-Evidence From A Factorial Survey.” Research in Social Stratification and Mobility46(Part B):112–28.
20.
GelmanAndrewHillJennifer. 2007. Data Analysis Using Regression and Multilevel/Hierarchical Models (Analytical Methods for Social Research). New York: Cambridge University Press.
21.
GreenPaul E.KriegerAbba M.WindYoram. 2001. “Thirty Years of Conjoint Analysis: Reflections and Prospects.” Interfaces31(3):S56–73.
22.
GreenPaul E.RaoVithala R.. 1971. “Conjoint Measurement for Quantifying Judgmental Data.” Journal of Marketing Research8(3):355–63.
23.
GreenPaul E.SrinivasanVenkat. 1990. “Conjoint Analysis in Marketing: New Developments with Implications for Research and Practice.” The Journal of Marketing54(4):3–19.
24.
GustafssonAndersHerrmannAndreasHuberFrank. 2000. “Conjoint Analysis as an Instrument of Market Research Practice.” Pp. 5–45inConjoint Measurement, edited byGustfssonA.HermannA.HuberF.. Berlin, Heidelberg: Springer.
25.
HuLi-tzeBentlerPeter M.. 1999. “Cutoff Criteria for Fit Indexes in Covariance Structure Analysis: Conventional Criteria Versus New Alternatives.” Structural Equation Modeling: A Multidisciplinary Journal6(1):1–55.
26.
JassoGuillermina. 2006. “Factorial Survey Methods for Studying Beliefs and Judgments.” Sociological Methods & Research34(3):334–423.
JassoGuillerminaOppKarl-Dieter. 1997. “Probing the Character of Norms: A Factorial Survey Analysis of the Norms of Political Action.” American Sociological Review62(6):947–64.
29.
KarrenRonald J.BarringerMelissa Woodard. 2002. “A Review and Analysis of the Policy-Capturing Methodology in Organizational Research: Guidelines for Research and Practice.” Organizational Research Methods5(4):337–61.
30.
LiebeUlfMoumouniIsmaïl M.BiglerChristineIngabireChantalBieriSabin. 2020. “Using Factorial Survey Experiments to Measure Attitudes, Social Norms, and Fairness Concerns in Developing Countries.” Sociological Methods & Research49(1):161–92.
31.
LyonsChristopher J. 2008. “Individual Perceptions and the Social Construction of Hate Crimes: A Factorial Survey.” The Social Science Journal45(1): 107–31.
MertzWilliam H.DohertyMichael E.. 1974. “The Influence of Task Characteristics on Strategies of Cue Combination.” Organizational Behavior and Human Performance12(2):196–216.
34.
PodsakoffPhilip M.MacKenzieScott B.LeeJeong-YeonPodsakoffNathan P.. 2003. “Common Method Biases in Behavioral Research: A Critical Review of the Literature and Recommended Remedies.” Journal of Applied Psychology88(5):879–903.
35.
RosseelYves. 2012. “lavaan: An R Package for Structural Equation Modeling.” Journal of Statistical Software48(2):1–36.
36.
RossiPeter H.AndersonAndy B.. 1982. “The Factorial Survey Approach: An Introduction.” Pp. 15–67inMeasuring Social Judgments: The Factorial Survey Approach, edited byRossiP. H.NockS. S.. Beverly Hills: Sage.
37.
SchwappachDavid L. B.GehringKatrin. 2014. “Silence That can Be Dangerous: A Vignette Study to Assess Healthcare Professionals’ Likelihood of Speaking Up About Safety Concerns.” PLoS One9(8):e104720.
38.
SchwartzShalom H.CieciuchJanVecchioneMicheleDavidovEldadFischerRonaldBeierleinConstanzeRamosAliceVerkasaloMarkkuLönqvistJan-Erik. 2012. “Refining the Theory of Basic Individual Values.” Journal of Personality and Social Psychology103(4):663–88. doi: 10.1037/a0029393
39.
ShamonHawalDülmerHermannGizaAdam. Forthcoming. “The Factorial Survey: The Impact of the Presentation Format of Vignettes on Answer Behavior and Processing Time.” Sociological Methods & Research. https://doi.org/10.1177/0049124119852382.
40.
ShlayAnne B. 2010. “African American, White and Hispanic Child Care Preferences: A Factorial Survey Analysis of Welfare Leavers by Race and Ethnicity.” Social Science Research39(1):125–41.
41.
SuDanSteinerPeter M.. 2020. “An Evaluation of Experimental Designs for Constructing Vignette Sets in Factorial Surveys.” Sociological Methods & Research49(2):455–97.
42.
TomassettiAlan J.DalalReeshad S.KaplanSeth A.. 2016. “Is Policy Capturing Really More Resistant Than Traditional Self-Report Techniques to Socially Desirable Responding?” Organizational Research Methods19(2):255–85.
43.
WallanderLisa. 2009. “25 Years of Factorial Surveys in Sociology: A Review.” Social Science Research38(3):505–20.
44.
WeijtersBertBaumgartnerHans. 2019. “Analyzing Policy Capturing Data Using Structural Equation Modeling for Within-Subject Experiments (SEMWISE).” Organizational Research Methods22(3):623–48.
45.
ZercherFlorianSchmidtPeterCieciuchJanDavidovEldad. 2015. “The Comparability of the Universalism Value Over Time and Across Countries in the European Social Survey: Exact vs. Approximate Measurement Invariance.” Frontiers in Psychology6:733.
