Double Barreled Questions: An Analysis of the Similarity of Elements and Effects on Measurement Quality

Abstract

In double barreled questions (DBQs) respondents provide one answer to two questions. Assumptions how respondents treat DBQs and how DBQs impact measurement quality are tested in two randomized experiments. DBQs are compared with revisions in which one stimulus was retained while the other stimulus was skipped. The observed means and parameters when modeling latent variables differed among the versions. Metric and scalar measurement invariance was not given among the versions, and at least one single stimulus version was found to be associated with a higher validity. Response latencies did not differ among versions or respondents needed less time to respond to DBQs. The author concludes that respondents may understand the stimuli in a DBQ differently, and access one of them while disregarding the other, which can have an adverse effect on validity.

Keywords

Question wording double barreled questions validity comparability measurement invariance

References

Aichholzer, J., and E. Zeglovits. 2015. “Balancierte Kurzskala autoritärer Einstellungen (B-RWA-6).” Zusammenstellung sozialwissenschaftlicher Items und Skalen (ZIS). DOI: https://doi.org/10.6102/zis239.

Altemeyer, B. 1981. Right-Wing Authoritarianism. Winnipeg: University of Manitoba Press.

Bassili, J.N., and B.S. Scott. 1996. “Response Latency as a Signal to Question Problems in Survey Research.” Public Opinion Quarterly 60(3): 390–399. DOI: https://org/doi.10.1086/297760.

Beauducel, A., and W.W. Wittmann. 2005. “Simulation Study on Fit Indexes in CFA Based on Data With Slightly Distorted Simple Structure.” Structural Equation Modeling: A Multidisciplinary Journal 12(1): 41–75. DOI: https://doi.org/10.1207/s15328007sem1201_3.

Beierlein, C., F. Asbrock, M. Kauff, and P. Schmidt. 2014. Die Kurzskala Autoritarismus (KSA-3): Ein ökonomisches Messinstrument zur Erfassung dreier Subdimensionen autoritärer Einstellungen. (GESIS-Working Papers, 2014/35). Mannheim: GESIS – Leibnitz-Institut für Sozialwissenschaften. Available at: http://www.gesis.org/fileadmin/kurzskalen/working_papers/KSA3_WorkingPapers_2014-35.pdf (accessed February 2019).

Bless, H., G. Bohner, T. Hild, and N. Schwarz. 1992. “Asking Difficult Questions: Task Complexity Increases the Impact of Response Alternatives.” European Journal of Social Psychology 22(3): 309–312. DOI: https://doi.org/10.1002/ejsp.2420220309.

Borgers, N., and J. Hox. 2001. “Item Nonresponse in Questionnaire Research with Children.” Journal of Official Statistics 17(2): 321–335. Available at: https://www.scb.se/contentassets/ca21efb41fee47d293bbee5bf7be7fb3/item-nonresponse-in-questionnaire-research-with-children.pdf (accessed September 2020).

Bradburn, N.M., S. Sudman, and B. Wansink. 2004. Asking Questions: The Definitive Guide to Questionnaire Design: For Market Research, Political Polls, and Social and Health Questionnaires. San Francisco: Jossey-Bass.

Campbell, J.C., D.W. Webster, and N. Glass. 2009. “The Danger Assessment: Validation of a Lethality Risk Assessment Instrument for Intimate Partner Femicide.” Journal of Interpersonal Violence 24(4): 653–674. DOI: https://doi.org/10.1177/08862605083doi.17180.

10.

Chen, F.F. 2007. “Sensitivity of Goodness of Fit Indexes to Lack of Measurement Invariance.” Structural Equation Modeling: A Multidisciplinary Journal 14(3): 464–504. DOI: https://doi.org/10.1080/10705510701301834.

11.

Cohen, J. 1992. “A Power Primer.” Psychological Bulletin 112(1): 155–159. DOI: https://doi.org/10.1037/0033-2909.112.1.155.

12.

Dillman, D.A., J.D. Smyth, and L.M. Christian. 2014. Internet, Mail, and Mixed-Mode Surveys: The Tailored Design Method. Hoboken, NJ: Wiley.

13.

ESS, European Social Survey. 2014. ESS Round 7 Source Questionnaire. London: ESS ERIC Headquarters, Centre for Comparative Social Surveys, City University London. Available at: https://www.europeansocialsurvey.org/docs/round7/fieldwork/source/ESS7_source_main_questionnaire.pdf (accessed February 2019).

14.

Fowler, F.J., Jr.. 1992. “How Unclear Terms Affect Survey Data.” Public Opinion Quarterly 56(2): 218–231. DOI: https://doi.org/10.1086/269312.

15.

Gemenis, K. 2013. “Estimating Parties’ Policy Positions through Voting Advice Applications. Some Methodological Considerations.” Acta Politica 48(3): 268–295. DOI: https://doi.org/10.1057/ap.2012.36.

16.

Graesser, A.C. 2006. “Question Understanding Aid (QUAID): A Web Facility that Tests Question Comprehensibility.” Public Opinion Quarterly 70(1): 3–22. DOI: https://doi.org/10.1093/poq/nfj012.

17.

Grant Levy, S. 2019. “Deconstructing a Double-Barreled Alternative: Evolution and Creationism.” Psychological Reports 122(5): 1995–2004. DOI: https://doi.org/10.1177/0033294118795145.

18.

Groves, R.M., F.J. Fowler, Jr., M.P. Cooper, J.M. Lepkowski, E. Singer, and R. Tourangeau. 2009. Survey Methodology, (2nd ed.). Oxford: Wiley.

19.

Hox, J.J., E.D. de Leeuw, and E.A.O. Zijlmans. 2015. “Measurement Equivalence in Mixed Mode Surveys.” Frontiers in Psychology 6: 87. DOI: https://doi.org/10.3389/fpsyg.2015.00087.

20.

Hu, L., and P.M. Bentler. 1999. “Cutoff Criteria for Fit Indexes in Covariance Structure Analysis: Conventional Criteria Versus New Alternatives.” Structural Equation Modeling: A Multidisciplinary Journal 6(1): 1–55. DOI: https://doi.org/10.1080/10705519909540118.

21.

Kane, M.T. 2013. “Validating the Interpretations and Uses of Test Scores.” Journal of Educational Measurement 50(1): 1–73. DOI: https://doi.org/10.1111/jedm.12000.

22.

Kay, P., and C.J. Fillmore. 1999. “Grammatical Constructions and Linguistic Generalizations: The What’s X Doing Y? Construction.” Language 75(1): 1–31. DOI: https://doi.org/10.2307/417472.

23.

Krosnick, J.A. 1991. “Response Strategies for Coping with the Cognitive Demands of Attitude Measures in Surveys.” Applied Cognitive Psychology 5(3): 213–216. DOI: https://doi.org/10.1002/acp.2350050305.

24.

Krosnick, J.A., and S. Presser. 2009. “Question and Questionnaire Design.” In Handbook of Survey Research, edited by J.D. Wright and P.V. Marsden., (2nd ed.) (pp. 263–313). San Diego, CA: Elsevier.

25.

Le Payne, S. 1951. The Art of Asking Questions. Princeton, NJ: Princeton University Press.

26.

Lenzner, T., C. Neuert, P. Hadler, A. Stiegler, C. Beitz, R. Schmidt, Z. Umuc, N. Reisepatt, S. Andrea, and N. Menold. 2017. “Krankheitswissen und Informationsbedarfe – Diabetes mellitus. Fragebogen für Personen mit Diabetes.” Available at: http://pretest.gesis.org/pdf/ProjektBericht/Projektbericht-17-03.pdf (accessed February 2019)

27.

Lenzner, T., L. Kaczmirek, and A. Lenzner. 2010. “Cognitive Burden of Survey Questions and Response Times: A Psycholinguistic Experiment.” Applied Cognitive Psychology 24(7): 1003–1020. DOI: https://doi.org/10.1002/acp.1602.

28.

Lord, F.M., and M.R. Novick. 1968. Statistical Theories of Mental Test Scores. Reading: Addison-Wesley.

29.

Menold, N., and T. Raykov. 2015. “Can Reliability of Multiple Component Measuring Instruments Depend on Response Option Presentation Mode?” Educational and Psychological Measurement 76(3): 454–569. DOI: https://doi.org/10.1177/0013164415593602.

30.

Meredith, W. 1993. “Measurement Invariance, Factor Analysis and Factorial Invariance.” Psychometrika 58(4): 525–543. DOI: https://doi.org/10.1007/BF02294825.

31.

Messick, S. 1989. “Meaning and Values in Test Validation: The Science and Ethics of Assessment.” Educational Researcher 18(2): 5–11. DOI: https://doi.org/10.3102/0013189X018002005.

32.

Muthén, B.O. 2002. “Beyond SEM: General latent variable modeling.” Behaviormetrika 29(1): 81–117. DOI: https://doi.org/10.2333/bhmk.29.81.

33.

Muthén, B.O., and T. Asparouhov. 2014. “IRT Studies of Many Groups: The Alignment Method.” Frontiers in Psychology 5,978. DOI: https://doi.org/10.3389/fpsyg.2014.00978.

34.

Muthén, L.K., and B.O. Muthén. 2017. Mplus User’s Guide. Los Angeles, CA: Muthén & Muthén.

35.

Olson, K. 2008. “Double Barreled Question.” In Encyclopedia of Survey Research Methods, edited by P.J. Lavrakas. (pp. 209–211). Thousand Oaks, CA: Sage Publications. DOI: https://doi.org/10.4135/9781412963947.n145.

36.

Oppenheim, A.N. 1992. Questionnaire Design, Interviewing, and Attitude Measurement. New York City: St. Martin’s Press.

37.

Raykov, T., and G.A. Marcoulides. 2011. Introduction to Psychometric Theory. New York: Taylor & Francis.

38.

Raykov, T. 2012. “Scale Construction and Development Using Structural Equation Modeling.” In Handbook of Structural Equation Modeling, edited by R.H. Hoyle. (pp. 472–492). New York: The Guilford Press.

39.

Raykov, T., N. Menold, and G.A. Marcoulides. 2018. “Studying Latent Criterion Validity for Complex Structure Measuring Instruments Using Latent Variable Modeling.” Educational and Psychological Measurement 78(5): 905–917. DOI: https://doi.org/10.1177/0013164417698017.

40.

Reise, S. 2012. “The rediscovery of bifactor measurement models.” Multivariate Behavioral Research 47: 667–696. DOI: https://doi.org/10.1080/00273171.2012.715555.

41.

Schaeffer, N.C., and J. Dykema. 2011. “Questions for Surveys: Current Trends and Future Directions.” Public Opinion Quarterly 75(5): 909–961. DOI: https://doi.org/10.1093/-poq/nfr048.

42.

Schwartz, S.H., B. Breyer, and D. Danner. 2015. “Human Values Scale (ESS).” Zusammenstellung sozialwissenschaftlicher Items und Skalen (ZIS). DOI: https://doi.org/10.6102/zis234.

43.

Schwartz, S.H. 2003. “A Proposal for Measuring Value Orientations across Nations.” In Questionnaire Development Package of the European Social Survey. Available at: http://www.europeansocialsurvey.org/docs/methodology/core_ess_questionnaire/ESS_core_questionnaire_human_values.pdf (accessed February 2019).

44.

Stafford, L. 2011. “Measuring relationship maintenance behaviors. Critique and development of the revised relationship maintenance behavior scale.” Journal of Social and Personal Relationships 28(2): 278–303. DOI: https://doi.org/10.1177/0265407510378125.

45.

Tourangeau, R., L.J. Rips, and K.A. Rasinski. 2000. The Psychology of Survey Response. Cambridge: Cambridge University Press.

46.

Vettehen, P.G.H., and L.B. van Snippenburg. 2002. “Measuring Motivations for Media Exposure: A Thesis.” Quality and Quantity 36(3): 259–276. DOI: https://doi.org/10.1023/A:1016076505379.

47.

Williams, R.T., A.W. Heinemann, R.K. Bode, C.S. Wilson, J.R. Fann, and D.G. Tate. 2009. “Improving Measurement Properties of the Patient Health Questionnaire–9 with Rating Scale Analysis.” Rehabilitation Psychology 54(2): 198–203. DOI: https://doi.org/10.1037/a0015529.

48.

Yan, T., and R. Tourangeau. 2008. “Fast times and easy questions: the effects of age, experience and question complexity on web survey response times.” Applied Cognitive Psychology 22(1): 51–68. DOI: https://doi.org/10.1002/acp.1331.

49.

Yorkston, K.M., C.R. Baylor, J. Dietz, B.J. Dudgeon, T. Eadie, R.M. Miller, and D. Amtmann. 2008. “Developing a Scale of Communicative Participation: A Cognitive Interviewing Study.” Disability and Rehabilitation 30(6): 425–433. DOI: https://doi.org/10.1080/09638280701625328.