Measuring attitudes and opinions employing agree/disagree (A/D) questions is a common method in social research because it appears to be possible to measure different constructs with identical response scales. However, theoretical considerations suggest that A/D questions require a considerable cognitive processing. Item-specific (IS) questions, in contrast, offer content-related response categories, implying less cognitive processing. To investigate the respective cognitive effort and response quality associated with A/D and IS questions, we conducted a web-based experiment with 1,005 students. Cognitive effort was assessed by response times and answer changes. Response quality, in contrast, was assessed by different indicators such as dropouts. According to our results, single IS questions require higher cognitive effort than single A/D questions in terms of response times. Moreover, our findings show substantial differences in processing single and grid questions.
Get full access to this article
View all access options for this article.
References
1.
BaddeleyA.1992. Working memory. Science255:556–59.
2.
BassiliJ. N.1996. How and why of response time latency measurement in telephone surveys. In Answering questions: Methodology for determining cognitive and communicative process in survey research, eds. SchwarzN.SudmanS., 319–46. San Francisco: Jossey-Bass.
3.
BassiliJ. N.FletscherJ. F.1991. Response time measurement in survey research. A method for CATI and a new look at non-attitudes. Public Opinion Quarterly55:331–46.
4.
BassiliJ. N.ScottB. S.1996. Response latency as a signal to question problems in survey research. Public Opinion Quarterly60:390–99.
5.
CarpenterP. A.JustM. A.1975. Sentence comprehension: A psycholinguistic processing model of verification. Psychological Review82:45–73.
6.
ChangL.KrosnickJ. A.2009. National surveys via RDD telephone interviewing versus the Internet: Comparing sample representativeness and response quality. Public Opinion Quarterly73:641–78.
7.
CohenJ.1969. Statistical power analysis for the behavioral science. New York: Academic Press.
8.
ConverseJ. M.PresserS.1986. Survey questions: Handcrafting the standardized questionnaire. Beverly Hills, CA: Sage.
9.
CouperM. P.KreuterF.2013. Using paradata to explore item level response times in surveys. Journal of the Royal Statistical Society176:271–86.
10.
CouperM. P.TourangeauR.ConradF. G.CrawfordS. D.2004. What they see is what we get: Response options for web surveys. Social Science Computer Review22:111–27.
11.
FazioR. H.1990. A practical guide to the use of response latency in social psychological research. In Research methods in personality and social research, eds. HendrickC.ClarkM. S., 74–97. Newbury Park, CA: Sage.
GalesicM.BosnjakM.2009. Effects of questionnaire length on participation and indicators of response quality in a web survey. Public Opinion Quarterly73:349–60.
HeerweghD.2011. Internet survey paradata. In Social and behavioral research and the Internet: Advances in applied methods and research strategies, eds. DasM.EsterP.KaczmirekL., 325–48. New York: Routledge.
17.
HoaglinD. C.MostellerF.TukeyJ. W.2000. Understanding robust and exploratory data analysis. New York: John Wiley.
18.
HöhneJ. K.LenznerT.2015. Investigating response order effects in web surveys using eye tracking. Psihologija48:361–77.
19.
KrosnickJ. A.1991. Response strategies for coping with the demands of attitude measures in surveys. Applied Cognitive Psychology5:213–36.
20.
KrosnickJ. A.PresserS.2010. Question and questionnaire design. In Handbook of survey research, eds. MarsdenP. V.WrightJ. D., 263–313. Bingley, UK: Emerald.
21.
LelkesY.WeissR.2015. Much ado about acquiescence: The relative validity and reliability of construct-specific and agree-disagree questions. Research and Politics. http://journals.sagepub.com/doi/pdf/10.1177/2053168015604173 (accessed February 3, 2017).
22.
LenznerT.KaczmirekL.LenznerA.2010. Cognitive burden of survey questions and response times: A psycholinguistic experiment. Applied Cognitive Psychology24:1003–20.
23.
RatcliffR.1993. Methods for dealing with reaction time outliers. Psychological Bulletin114:510–32.
24.
SarisW.RevillaM.KrosnickJ. A.ShaefferE. M.2010. Comparing questions with agree/disagree response options to questions with item-specific response options. Survey Research Methods4:61–79.
SchumanH.PresserS.1996. Questions and answers in attitude surveys. Experiments on question form, wording, and context. Thousand Oaks, CA: Sage.
27.
SternM. J.2008. The use of client-side paradata in analyzing the effects of visual layout on changing responses in web surveys. Field Methods20:377–98.
28.
YanT.OlsonK.2013. Analyzing paradata to investigate measurement error. In Improving surveys with paradata: Analytic uses of process information, ed. KreuterF., 73–96. Hoboken, NJ: John Wiley.
29.
YanT.TourangeauR.2008. Fast times and easy questions. The effects of age, experience, and question complexity on web survey response times. Applied Cognitive Psychology22:51–68.
