Knowledge questions frequently are used in survey research to measure respondents’ topic-related cognitive ability and memory. However, in self-administered surveys, respondents can search external sources for additional information to answer a knowledge question correctly. In this case, the knowledge question measures accessible and procedural memory. Depending on what the knowledge question aims at, the validity of this measure is limited. Thus, in this study, we conducted three experiments using a web survey to investigate the effects of task difficulty, respondents’ ability, and respondents’ motivation on the likelihood of searching external sources for additional information as a form of over-optimizing response behavior when answering knowledge questions. We found that the respondents who are highly educated and more interested in a survey are more likely to invest additional efforts to answer knowledge questions correctly. Most importantly, our data showed that for these respondents, a more difficult question design further increases the likelihood of over-optimizing response behavior.
BerensFlorianSchlosserStephanHöhneJan K.. 2018. “On-device and Off-device Multitasking and Their Effects on Web and Mobile Web Survey Completion.” Paper presented at the General Online Research Conference (GOR 18), March 01, 2018, Cologne, Germany.
2.
CallegaroMarioDiSograCharles. 2008. “Computing Response Metrics for Online Panels.” Public Opinion Quarterly72:1008–32.
3.
CliffordScottJeritJennifer. 2016. “Cheating on Political Knowledge Questions in Online Surveys: An Assessment of the Problem and Solutions.” Public Opinion Quarterly80:858–87.
4.
CouperMick P.PetersonGregg J.. 2017. “Why Do Web Surveys Take Longer on Smartphones?” Social Science Computer Review35:357–77.
5.
De VreeseClaes H.BoukesMarkSchuckAndreasVliegenthartRensBosLindaLelkesYph. 2017. “Linking Survey and Media Content Data: Opportunities, Considerations, and Pitfalls.” Communication Methods and Measures11:221–44.
6.
Delli CarpiniMichael X.KeeterScott. 1996. What Americans Know about Politics and Why It Matters. New Haven, MA: Yale University Press.
7.
DiedenhofenBirkMuschJochen. 2017. “Pagefocus: Using Paradata to Detect and Prevent Cheating on Online Achievement Tests.” Behavior Research Methods49:1444–59.
8.
DillmanDon A.SmythJolene D.ChristianLeah M.. 2014. Internet, Phone, Mail, and Mixed-method Surveys: The Tailored Design Method. Hoboken, NJ: Wiley.
9.
GalstonWilliam A.2001. “Political Knowledge, Political Engagement, and Civic Education.” Annual Review of Political Science4:217–34.
10.
GreszkiRobertMeyerMarcoSchoenHarald. 2015. “Exploring the Effects of Removing ‘Too Fast.’ Responses and Respondents from Web Surveys.” Public Opinion Quarterly79:471–503.
11.
GummerTobiasQuoßFranziskaRoßmannJoss. 2019. “Does Increasing Mobile Device Coverage Reduce Heterogeneity in Completing Web Surveys on Smartphones?” Social Science Computer Review37:371–84.
12.
GummerTobiasRoßmannJossSilberHenning. 2018. “Using Instructed Response Items as Attention Checks in Web Surveys: Properties and Implementation.” Sociological Methods & Research. https://doi.org/10.1177/0049124118769083.
13.
HöhneJan K.SchlosserStephan. 2018. “Investigating the Adequacy of Response Time Outlier Definitions in Computer-based Web Surveys Using Paradata Surveyfocus.” Social Science Computer Review36:369–78.
14.
HolbrookAllyson L.KrosnickJon A.MooreDavidTourangeauRoger. 2007. “Response Order Effects in Dichotomous Categorical Questions Presented Orally: The Impact of Question and Respondent Attributes.” Public Opinion Quarterly71:325–48.
15.
JensenCarstenThomsenJens P. F.. 2014. “Self-reported Cheating in Web Surveys on Political Knowledge.” Quality & Quantity48:3343–54.
16.
KleinerBrianLippsOliverFerrezEliane. 2015. “Language Ability and Motivation among Foreigners in Survey Responding.” Journal of Survey Statistics and Methodology3:339–60.
17.
KnäuperBärbel. 1999. “The Impact of Age and Education on Response Order Effects in Attitude Measurement.” Public Opinion Quarterly63:347–70.
18.
KrosnickJon A.1991. “Response Strategies for Coping with the Cognitive Demands of Attitude Measures in Surveys.” Applied Cognitive Psychology5:213–36.
19.
KrosnickJon A.1999. “Survey Research.” Annual Review of Psychology50:537–67.
20.
KrosnickJon A.PresserStanley. 2010. “Question and Questionnaire Design.” Pp. 263–313 in Handbook of Survey Research, edited byMarsdenP. V.WrightJ. D.. Bingley, England: Emerald Group.
21.
LenznerTimo. 2012. “Effects of Survey Question Comprehensibility on Response Quality.” Field Methods24:409–28.
22.
LenznerTimoKaczmirekLarsGalesicMirta. 2011. “Seeing through the Eyes of the Respondent: An Eye-tracking Study on Survey Question Comprehension.” International Journal of Public Opinion Research23:361–73.
23.
LenznerTimoKaczmirekLarsLenznerAlwine. 2010. “Cognitive Burden of Survey Questions and Response Times: A Psycholinguistic Experiment.” Applied Cognitive Psychology24:1003–20.
24.
MavletovaAigulCouperMick P.. 2016. “Grouping of Items in Mobile Web Questionnaires.” Field Methods28:170–93.
25.
MondakJeffery J.DavisBelinda C.. 2001. “Asked and Answered: Knowledge Levels When We Will Not Take “Don’t Know” for an Answer.” Political Behavior23:199–224.
26.
NadeauRichardNiemiRichard G.. 1995. “Educated Guesses: The Process of Answering Factual Knowledge Questions in Surveys.” Public Opinion Quarterly59:323–46.
27.
NarayanSowmyaKrosnickJon A.. 1996. “Education Moderates Some Response Effects in Attitude Measurement.” Public Opinion Quarterly60:58–88.
28.
PaulhusDelroy L.2002. “Socially Desirable Responding: The Evolution of a Construct.” Pp. 49–69 in The Role of Constructs in Psychological and Educational Measurement, edited byBraunH. I.JacksonD. N.. Mahwah, NJ: Erlbaum.
29.
PriorMarkus. 2005. “News vs. Entertainment: How Increasing Media Choice Widens Gaps in Political Knowledge and Turnout.” American Journal of Political Science49:577–92.
30.
PriorMarkusLupiaArthur. 2008. “Money, Time, and Political Knowledge: Distinguishing Quick Recall and Political Learning Skills.” American Journal of Political Science52:169–83.
31.
RejaUršaManfredaKatja LozarHlebecValentinaVehovarVasja. 2003. “Open-ended vs. Close-ended Questions in Web Questionnaires.” Developments in Applied Statistics19:159–77.
32.
RobisonJoshua. 2015. “Who Knows? Question Format and Political Knowledge.” International Journal of Public Opinion Research27:1–21. doi:10.1093/ijpor/edu019
33.
RoßmannJossGummerTobiasSilberHenning. 2017. “Mitigating Satisficing in Cognitively Demanding Grid Questions: Evidence from Two Web-based Experiments.” Journal of Survey Statistics and Methodology6:376–400.
34.
ScharkowMichael. 2016. “The Accuracy of Self-reported Internet Use—A Validation Study Using Client Log Data.” Communication Methods and Measures10:13–27.
SendelbahAnžeVehovarVasjaSlavecAnaAndražPetrovčič.. 2016. “Investigating Respondent Multitasking in Web Surveys Using Paradata.” Computers in Human Behavior55:777–87.
37.
SmythJolene D.DillmanDon A.ChristianLeah M.McBrideMallory. 2009. “Open-ended Questions in Web Surveys: Can Increasing the Size of Answer Boxes and Providing Extra Verbal Instructions Improve Response Quality?” Public Opinion Quarterly73:325–37.
38.
ToepoelVeraDasMarcelVan SoestArthur. 2009. “Design of Web Questionnaires: The Effects of the Number of Items Per Screen.” Field Methods21:200–13.
39.
TourangeauRogerRipsLance J.RasinskiKenneth. 2000. The Psychology of Survey Response. Cambridge, UK: Cambridge University Press.
40.
TourangeauRogerYanTing. 2007. “Sensitive Questions in Surveys.” Psychological Bulletin133:859–83.
41.
YanTingTourangeauRoger. 2008. “Fast Times and Easy Questions: The Effects of Age, Experience, and Question Complexity on Web Survey Response Times.” Applied Cognitive Psychology22:51–68.
42.
ZwarunLaraHallAlice. 2014. “What’s Going On? Age, Distraction, and Multitasking During Online Survey Taking.” Computers in Human Behavior41:236–44.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
