The factorial survey is an experimental design consisting of varying situations (vignettes) that have to be judged by respondents. For more complex research questions, it quickly becomes impossible for an individual respondent to judge all vignettes. To overcome this problem, random designs are recommended most of the time, whereas quota designs are not discussed at all. First comparisons of random designs with fractional factorial and D-efficient designs are based on fictitious data, first comparisons with fractional factorial and confounded factorial designs are restricted to theoretical considerations. The aim of this contribution is to compare different designs regarding their reliability and their internal validity. The benchmark for the empirical comparison is established by the estimators from a parsimonious full factorial design, each answered by a sample of 132 students (real instead of fictitious data). Multilevel analyses confirm that, if they exist, balanced confounded factorial designs are ideal. A confounded D-efficient design, as proposed for the first time in this article, is also superior to simple random designs.
AddelmanSidney. 1962. “Orthogonal Main-effect Plans for Asymmetrical Factorial Experiments.” Technometrics4:21–46.
2.
AlexanderCheryl S.BeckerHenry J.. 1978. “The Use of Vignettes in Survey Research.” Public Opinion Quarterly42:93–104.
3.
AlvesWeyne M.RossiPeter H.. 1978. “Who Should Get What? Fairness Judgments of the Distributive Earnings.” American Journal of Sociology84:541–64.
4.
AtzmüllerChristianeSteinerPeter M.. 2010. “Experimental Vignette Studies in Survey Research.” Methodology6:128–38.
5.
BackhausKlausErichsonBerndPlinkeWulffWeiberRolf. 2000. Multivariate Analysemethoden. Eine anwendungsorientierte Einführung. Berlin, Germany: Springer.
6.
BeckMichaelOppKarl-Dieter. 2001. “Der faktorielle Survey und die Messung von Normen.” Kölner Zeitschrift für Soziologie und Sozialpsychologie53:283–306.
7.
BürklinWilhelmKleinMarkusRußAchim. 1996. “Postmaterialistischer oder anthropozentrischer Wertewandel? Eine Erwiderung auf Ronald Inglehart und Hans-Dieter Klingemann.” Politische Vierteljahresschrift37:517–36.
8.
DülmerHermann. 2007. “Experimental Plans in Factorial Surveys: Random or Quota Design?” Sociological Methods & Research35:382–409.
9.
FoxJohn. 1991. Regression Diagnostics: An Introduction. Newbury Park, CA: Sage.
10.
FriedrichRobert J.1982. “In Defense of Multiplicative Terms in Multiple Regression Equations.” American Journal of Political Science26:797–833.
11.
GunstRichardMasonRobert L.. 1991. How to Construct Fractional Factorial Experiments. Vol. 14. The Basic References in Quality Control: Statistical Techniques. Milwaukee, WI: ASQC Quality Press.
12.
HoxJoop J.2010. Multilevel Analysis. Techniques and Applications. 2nd ed. New York: Routledge.
13.
HoxJoop J.KreftIta G. G.HermkensPiet L. J.. 1991. “The Analysis of Factorial Surveys.” Sociological Methods & Research19:493–510.
14.
InglehartRonald. 1977. The Silent Revolution. Changing Values and Political Styles Among Western Publics. Princeton, NJ: Princeton University Press.
15.
InglehartRonald. 1994. “‘Polarized Priorities or Flexible Alternatives? Dimensionality in Inglehart’s Materialism-postmaterialism Scale’: A Comment.” International Journal of Public Opinion Research6:289–92.
16.
InglehartRonald. 1997. Modernization and Postmodernization. Cultural, Economic, and Political Change in 43 Societies. Princeton, NJ: Princeton University Press.
17.
JassoGuillermina. 2006. “Factorial Survey Methods for Studying Beliefs and Judgments.” Sociological Methods & Research34:334–423.
18.
JassoGuillerminaRossiPeter H.. 1977. “Distributive Justice and Earned Income.” American Sociological Review42:639–51.
19.
KirkRoger E.1995. Experimental Design: Procedures for the Behavioral Sciences. 3rd ed. Pacific Grove, CA: Brooks/Cole Publishing Company.
20.
KleinMarkusDülmerHermannOhrDieterQuandtMarkusRosarUlrich. 2004. “Response Sets in the Measurement of Values: A Comparison of Rating and Ranking Procedures.” International Journal of Public Opinion Research16:474–83.
21.
KmentaJan. 1971. Elements of Econometrics. New York: Macmillan.
KuhfeldWarren F.2010. “Experimental Design: Efficiency, Coding, and Choice Designs.” Pp. 53–241 in Marketing Research Methods in SAS. Experimental Design, Choice, Conjoint, and Graphical Techniques, edited byKuhfeldW. F.. Retrieved October 8, 2013 (http://support.sas.com/resources/papers/tnote/tnote_marketresearch.html).
LawsonJohn. 2002. “Regression Analysis of Experiments with Complex Confounding Pattern Guided by the Alias Matrix.” Computational Statistics & Data Analysis39:227–41.
26.
LouviereJordan J.1988. Analyzing Decision Making. Metric Conjoint Analysis. Sage University Paper Series on Quantitative Applications in Social Sciences, 07-067. Newbury Park, CA: Sage.
27.
McLeanRobert A.AndersonVirgil A.. 1984. Applied Factorial and Fractional Designs. New York: Marcel Dekker.
28.
NeumanW. Lawrence. 2006. Social Research Methods. Qualitative and Quantitative Approaches. 6th ed. Boston, MA: Pearson.
29.
NockSteven L.1982. “Family Social Status: Consensus on Characteristics.” Pp. 95–118 in Measuring Social Judgments. The Factorial Survey Approach, edited byRossiP. H.NockSt. L.. Beverly Hills, CA: Sage.
30.
RaudenbushStephen W.BrykAnthony S.. 2002. Hierarchical Linear Models. Applications and Data Analysis Methods. 2nd ed. Thousand Oaks, CA: Sage.
31.
RossiPeter H.1979. “Vignette Analysis. Uncovering the Normative Structure of Complex Judgments.” Pp. 176–86 in Qualitative and Quantitative Social Research: Papers in Honor of Paul F. Lazarsfeld, edited byMertonR. K.ColemanJ. S.RossiP. H.. New York: The Free Press.
32.
RossiPeter H.AndersonAndy B.. 1982. “The Factorial Survey Approach: an Introduction.” Pp. 15–67 in Measuring Social Judgments. The Factorial Survey Approach, edited byRossiP. H.NockSt. L.. Beverly Hills, CA: Sage.
33.
RyanThomas P.2007. Modern Experimental Design. Hoboken, NJ: John Wiley & Sons.
34.
SahaiHardeoAgeelMohammed I.. 2000. The Analysis of Variance. Fixed, Random and Mixed Models. Boston, MA: Birkhäuser.
SnijdersTom A. B.BoskerRoel J.. 1994. “Modeled Variance in Two-level Models.” Sociological Methods & Research22:342–63.
37.
SnijdersTom A. B.BoskerRoel J.. 2012. Multilevel Analysis. An Introduction to Basic and Advanced Multilevel Modeling. 2nd ed. Los Angeles, CA: Sage.
38.
SteenbergenMarco R.JonesBradford S.. 2002. “Modeling Multilevel Data Structures.” American Journal of Political Science46:218–37.
39.
SteinerPeter M.AtzmüllerChristiane. 2006. “Experimentelle Vignettendesigns in faktoriellen Surveys.” Kölner Zeitschrift für Soziologie und Sozialpsychologie58:117–46.
40.
StevensStanley S.1975. Psychometrics. Introduction to its Perceptual, Neutral, and Social Prospects. New York: John Wiley & Sons.
41.
ThomeHelmut. 1990. “Grundkurs Statistik für Historiker. Teil II: Induktive Statistik und Regressionsanalyse.” Historical Social Research, Supplement No. 3. Cologne: University of Cologne.
42.
ThomeHelmut. 1991. “Modelling and Testing Interactive Relationships within Regression Analysis.” Historical Social Research16:21–50.
43.
WallanderLisa. 2009. “25 Years of Factorial Surveys in Sociology: A Review.” Social Science Research38:505–20.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
