In recent decades, factorial survey experiments (FSEs) have become increasingly widespread and successful for analyzing attitudes and behavioral intentions. FSEs measure the ratings of multidimensional treatments embedded in textual scenarios, which are called vignettes. Analyses of FSEs often assume that these ratings are interval scaled. Past research indicates that this assumption is problematic. Therefore, the following article develops a design for interval scaling tests in FSEs and a method for analyzing FSEs which is sensitive to the scaling level of ratings. An exemplary application of scaling sensitive factorial survey analysis in comparison to standard methods yields effect sizes, which are about a sixth larger with regard to the treatments used in the FSE and about a third larger with respect to influences of between respondent differences. The new method also enables the evaluation of interval in contrast to noninterval scaled rating behavior.
AllisonPaul D.2001. “Multiple Imputation for Missing Data: A Cautionary Tale.” Sociological Methods and Research28(3):301–09.
2.
AndersonNorman H.1976. “How Functional Measurement Can Yield Validated Interval Scales of Mental Quantities.” Journal of Applied Psychology61(6):677–92.
3.
AtzmüllerChristianeSteinerPeter M.. 2010. “Experimental Vignette Studies in Survey Research.” Methodology: European Journal of Research Methods for the Behavioral and Social Sciences6(3):128–38.
4.
AuspurgKatrinGundertStefanie. 2015. “Precarious Employment and Bargaining Power: Results of a Factorial Survey Analysis.” Zeitschrift für Soziologie44(2):99–117.
5.
AuspurgKatrinHinzThomas. 2015. Factorial Survey Experiments. Quantitative Applications in the Social Sciences 175. Thousand Oaks, CA: Sage.
6.
BirnbaumMichael H.1980. “Comparison of Two Theories of “Ratio” and “Difference” Judgments.” Journal of Experimental Psychology: General109(3):304–19.
7.
ChenChuanshengLeeShin-yingStevensonHarold W.. 1995. “Response Style and Cross-cultural Comparisons of Ranting Scales among East Asian and North American Students.” Psychological Science6(3):170–75.
8.
CoolsWalentinaHofmansJoeriTheunsPeter. 2006. “Context in Category Scales: Is “Fully Agree” Equal to Twice Agree?” European Review of Applied Psychology56(4):223–29.
9.
DülmerHermann. 2007. “Experimental Plans in Factorial Surveys: Random or Quota Design?” Sociological Methods and Research35(3):382–409.
10.
GroßMartinLangVolker. 2018. “Why Citizens Oppose Inheritance Taxes – Even if They Do Not Have to Pay Them: Results of a Factorial Survey.” Zeitschrift für Soziologie (Journal of Sociology)47(3):200–207.
11.
HardinCurtisBirnbaumMichael H.. 1990. “Malleability of “Ratio” Judgments of Occupational Prestige.” American Journal of Psychology103(1):1–20.
12.
HedekerDonald. 2008. “Multilevel Models for Ordinal and Nominal Variables.” Pp. 237–74 in Handbook of Multilevel Analysis, edited by LeeuwJan deMeijerErik. New York: Springer.
13.
HofmansJoeriTheunsPerterMairesseOliver. 2007. “Impact of the Number of Response Categories on Linearity and Sensitivity of Self-anchoring Scales: A Functional Measurement Approach.” Methodology3(4):160–69.
14.
HoxJoop J.2002. Multilevel Analysis: Techniques and Applications. New York: Routledge.
15.
HoxJoop J.KreftIta G. G.HermkensPiet L. J.. 1991. “The Analysis of Factorial Surveys.” Sociological Methods and Research19(4):493–510.
16.
JassoGuillermina. 2006. “Factorial Survey Methods for Studying Beliefs and Judgments.” Sociological Methods and Research34(3):334–423.
17.
JassoGuillermina. 1996. “Exploring the Reciprocal Relations between Theoretical and Empirical Work: The Case of the Justice Evaluation Function.” Sociological Methods and Research24(3):253–303.
18.
JassoGuillermina. 1978. “On the Justice of Earnings: A New Specification of the Justice Evaluation Function.” American Journal of Sociology83(6):1398–419.
19.
JassoGuillerminaRossiPeter H.. 1977. “Distributive Justice and Earned Income.” American Sociological Review42(4):639–51.
20.
JassoGuillerminaWegenerBernd. 1997. “Methods for Empirical Justice Analysis: Part 1. Framework, Models, and Quantities.” Social Justice Research10(4):393–430.
21.
JohnsonF. ReedKanninenBarbaraBinghamMatthewÖzdemirSemra. 2006. “Experimental Design for Stated-choice Studies.” Pp. 159–202 in Valuing Environmental Amenities Using Stated Choice Studies: A Common Sense Approach to Theory and Practice, edited by KanninenBarbara. Dordrecht, the Netherlands: Springer.
KendallMaurice G.1938. “A New Measure of Rank Correlation.” Biometrika30(1/2):81–93.
24.
KrantzDavid H.1972. “Measurement Structures and Psychological Laws.” Science175(4029):1427–35.
25.
KrantzDavid H.Duncan LuceR.SuppesPatrickTverskyAmos. 1971. Foundations of Measurement: Volume I: Additive and Polynomial Representations. London, England: Academic Press.
26.
KuhfeldWarren F.TobiasRandall D.GarrattMark. 1994. “Efficient Experimental Design with Marketing Research Applications.” Journal of Marketing Research31(4):545–57.
27.
LangVolker. 2016. “Determinants of Interval Scaled Response Behavior in Factorial Surveys.” Bielefeld, Germany: Bielefeld University, Working Paper.
28.
LiebigStefanSauerCarstenFriedhoffStefan. 2015. “Using Factorial Surveys to Study Justice Perceptions: Five Methodological Problems of Attitudinal Justice Research.” Social Justice Research28(4):415–34.
29.
LittleRoderick J. A.RubinDonald B.. 1989. “The Analysis of Social Science Data with Missing Values.” Sociological Methods and Research18(2-3):292–326.
30.
LouviereJordan J.HensherDavid A.SwaitJoffre D.. 2000. Stated Choice Methods: Analysis and Applications. Cambridge, England: Cambridge University Press.
31.
ManskiCharles F.1977. “The Structure of Random Utility Models.” Theory and Decision8(3):229–54.
32.
McFaddenDaniel. 1973. “Conditional Logit Analysis of Qualitative Choice Behavior.” Pp. 105–42 in Frontiers in Econometrics, edited by ZarembkaPaul. New York: Academic Press.
33.
MislevyRobert J.1991. “Randomization-based Inference about Latent Variables from Complex Samples.” Psychometrika56(2):177–96.
34.
OrthBernhard. 1982. “A Theoretical and Empirical Study of Scale Properties of Magnitude-estimation and Category-rating Scales.” Pp. 351–77 in Social Attitudes and Psychophysical Measurement, edited by WegenerBernd. Hillsdale, NJ: Lawrence Erlbaum.
35.
OrthBernhardWegenerBernd. 1983. “Scaling Occupational Prestige by Magnitude Estimation and Category Rating Methods: A Comparison with the Sensory Domain.” European Journal of Social Psychology13(4):417–31.
RammstedtBeatriceBeierleinConstanze. 2014. “Can’t We Make It Any Shorter? The Limits of Personality Assessment and Ways to Overcome Them.” Journal of Individual Differences35(4):212–20.
38.
RossiPeter H.AndersonAndy B.. 1982. “The Factorial Survey Approach: An Introduction.” Pp. 15–67 in Measuring Social Judgments: The Factorial Survey Approach, edited by RossiPeter H.NockSteven L.. Beverly Hills, CA: Sage.
39.
SchaefferNora C.BradburnNorman M.. 1989. “Respondent Behavior in Magnitude Estimation.” Journal of the American Statistical Association84(406):402–13.
40.
SimeanerHansRammMichaelKolbert-RammChrista. 2014. Datenalmanach Studierendensurvey 1993–2010: Studiensituation und Studierende an Universitäten und Fachhochschulen nach Geschlecht. Arbeitsgruppe Hochschulforschung Heft 75. Konstanz, Germany: University of Konstanz.
41.
Statistisches Bundesamt. 2014. Bildung und Kultur: Studierende an Hochschulen: Wintersemester 2013/2014. Fachserie 11, Reihe 4.1. Wiesbaden, Germany: Statistisches Bundesamt (Federal Office of Statistics).
42.
SwaitJoffre D.LouviereJordan J.. 1993. “The Role of the Scale Parameter in the Estimation and Use of Multinomial Logit Models.” Journal of Marketing Research30(3):305–14.
43.
TobinJames. 1958. “Estimation of Relationships for Limited Dependent Variables.” Econometrica26(1):24–36.
44.
WallanderLisa. 2009. “25 Years of Factorial Surveys in Sociology: A Review.” Social Science Research38(3):505–20.
45.
WegenerBernd, ed. 1982. Social Attitudes and Psychophysical Measurement. Hillsdale, NJ: Lawrence Erlbaum.
46.
WegenerBernd. 1983. “Category-rating and Magnitude Estimation Scaling Techniques: An Empirical Comparison.” Sociological Methods and Research12(1):31–75.
47.
WeinbergJill D.FreeseJeremyMcElhattenDavid. 2014. “Comparing Data Characteristics of an Online Factorial Survey between a Population-based and a Crowdsource-recruited Sample.” Sociological Science1:292–310.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
