Political knowledge is central to understanding citizens’ engagement with politics. Yet, as surveys are increasingly conducted online, participants’ ability to search the web may undermine the validity of factual knowledge measures. Recent research shows this search behavior is common, even when respondents are instructed otherwise. However, we know little about how outside search affects the validity of political knowledge measures. Using a series of experimental and observational studies, we provide consistent evidence that outside search degrades the validity of political knowledge measures. Our findings imply that researchers conducting online surveys need to take steps to discourage and diagnose search engine use.
AlthausScott L.2003. Collective Preferences in Democratic Politics: Opinion Surveys and the Will of the People. Cambridge: Cambridge University Press.
4.
AlwinDuane F.1991. “Family of Origin and Cohort Differences in Verbal Ability.” American Sociological Review56 (5): 625–38.
5.
AnsolabehereStephenRoddenJonathanSnyderJames M.2008. “The Strength of Issues: Using Multiple Measures to Gauge Preference Stability, Ideological Constraint, and Issue Voting.” American Political Science Review102 (2): 215–32.
6.
BerinskyAdam J.MargolisMichele F.SancesMichael W.2014. “Separating the Shirkers from the Workers? Making Sure Respondents Pay Attention on Self-Administered Surveys.” American Journal of Political Science58:739–53.
7.
BialekMichalPennycookGordon. 2018. “The Cognitive Reflection Test Is Robust to Multiple Exposures.” Behavior Research Methods50:1953–59.
8.
BullockJohn G.GerberAlan S.HillSeth J.HuberGregory A.2015. “Partisan Bias in Factual Beliefs about the Economy.” Quarterly Journal of Political Science10:519–78.
9.
BurnettCraig M.2016. “Exploring the Difference in Participants’ Factual Knowledge between Online and In-Person Survey Modes.” Research & Politics3 (2):120–31.
10.
CliffordScottJeritJennifer. 2014. “Is There a Cost to Convenience? An Experimental Comparison of Data Quality in Laboratory and Online Studies.” Journal of Experimental Political Science1:120–31.
11.
CliffordScottJeritJennifer. 2016. “Cheating on Political Knowledge Questions in Online Surveys.” Public Opinion Quarterly80 (4): 858–87.
12.
CarpiniDelliMichaelX.KeeterScott. 1996. What Americans Know about Politics and Why It Matters. New Haven: Yale University Press.
13.
DiedenhofenBirkMuschJochen. 2017. “PageFocus: Using Paradata to Detect and Prevent Cheating on Online Achievement Tests.”Behavior Research Methods49 (4): 1444–59.
14.
FedericoChristopher M.2004. “Predicting Attitude Extremity: The Interactive Effects of Schema Development and the Need to Evaluate and Their Mediation by Evaluative Integration.” Personality and Social Psychology Bulletin30 (10): 1281–94.
15.
FiskeSusan T.LauRichard R.SmithRichard A.1990. “On the Varieties and Utilities of Political Expertise.” Social Cognition8 (1): 31–48.
16.
GibsonJames L.CaldeiraGregory A.2015. “Knowing the Supreme Court? A Reconsideration of Public Ignorance of the High Court.” The Journal of Politics71:429–41. doi:10.1017/S0022381609090379.
17.
GoochAndrew. 2015. “Measurements of Cognitive Skill by Survey Mode: Marginal Differences and Scaling Similarities.” Research & Politics2 (3). doi:10.1177/2053168015590681.
18.
GoslingSamuel D.RentfrowPeter J.SwannWilliam B.2003. “A Very Brief Measure of the Big-Five Personality Domains.” Journal of Research in Personality37 (6): 504–28.
19.
HeerweghDirk. 2009. “Mode Differences between Face-to-Face and Web Surveys: An Experimental Investigation of Data Quality and Social Desirability Effects.” International Journal of Public Opinion Research21 (1): 111–21.
20.
HillygusD. SunshineJacksonNatalieYoungMcKenzie. 2014. “Professional Respondents in Non-probability Online Panels.” In Online Panel Research: A Data Quality Perspective, edited by CallegaroMarioBakerReginald P.BethlehemJelkeGöritzAnja S.KrosnickJon A.LavrakasPaul J., 219–36. Chichester: John Wiley.
21.
JacobyWilliam G.2006. “Value Choices and American Public Opinion.” American Journal of Political Science50 (3): 706–23.
22.
JensenCarstenThomsenJens Peter Frølund. 2014. “Self-Reported Cheating in Web Surveys on Political Knowledge.” Quality & Quantity48 (6): 3343–54.
23.
JeritJenniferBarabasJason. 2017. “Revisiting the Gender Gap in Political Knowledge.” Political Behavior39 (4): 817–38.
24.
KahanDan M.2016. “‘Ordinary Science Intelligence’: A Science-Comprehension Measure for Study of Risk and Science Communication, with Notes on Evolution and Climate Change.” Journal of Risk Research20:995–1016.
25.
KamCindy D.2005. “Who Toes the Party Line? Cues, Values, and Individual Differences.” Political Behavior27 (2): 163–82.
26.
KamCindy D.TrusslerMarc J.2016. “At the Nexus of Observational and Experimental Research: Theory, Specification, and Analysis of Experiments with Heterogeneous Treatment Effects.” Political Behavior39:789–815.
27.
KaneJohn V.BarabasJason. 2018. “No Harm in Checking: Using Factual Manipulation Checks to Assess Attentiveness in Experiments.” American Journal of Political Science63:234–49.
28.
KinderDonald R.2006. “Belief Systems Today.” Critical Review18 (1–3): 197–216.
29.
KinderDonald R.KalmoeNathan P.2017. Neither Liberal nor Conservative: Ideological Innocence in the American Public. Chicago: University of Chicago Press.
30.
KraftPatrick. 2019. Let’s Talk Politics: A Naive Approach for Measuring Political Sophistication. Working Paper, University of Wisconsin-Milwaukee.
31.
KreuterFraukePresserStanleyTourangeauRoger. 2008. “Social Desirability Bias in CATI, IVR, and Web Surveys: The Effects of Mode and Question Sensitivity.” Public Opinion Quarterly72 (5): 847–65.
32.
LauRichard R.RedlawskDavid P.2001. “Advantages and Disadvantages of Cognitive Heuristics in Political Decision Making.” American Journal of Political Science45 (4): 951–71.
33.
LizotteMary-KateSidmanAndrew H.2009. “Explaining the Gender Gap in Political Knowledge.” Politics & Gender5 (2): 127–51.
LuskinRobert C.BullockJohn G.2011. “‘Don’t Know’ Means ‘Don’t Know’: DK Responses and the Public’s Level of Political Knowledge.” The Journal of Politics73 (2): 547–57.
36.
MasonLilliana. 2018. “Ideologues without Issues: The Polarizing Consequences of Ideological Identities.” Public Opinion Quarterly 82 (Suppl. 1): 280–301.
37.
MillerMelissa K.OrrShannon K.2008. “Experimenting with a ‘Third Way’ in Political Knowledge Estimation.” Public Opinion Quarterly72 (4): 768–80.
38.
MondakJeffery J.AndersonMary R.2004. “The Knowledge Gap: A Reexamination of Gender-Based Differences in Political Knowledge.” The Journal of Politics66 (2): 492–512.
39.
MondakJeffrey J.2001. “Developing Valid Knowledge Scales.” American Journal of Political Science45 (1): 224–38.
40.
MontgomeryJacob M.NyhanBrendanTorresMichelle. 2018. “How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It.” American Journal of Political Science64:760–75.
41.
MottaMatthew P.CallaghanTimothy H.SmithBrianna. 2016. “Looking for Answers: Identifying Search Behavior and Improving Knowledge-Based Data Quality in Online Surveys.” International Journal of Public Opinion Research29 (4): 575–603.
42.
MunzertSimonSelbPeter. 2017. “Measuring Political Knowledge in Web-Based Surveys.”Social Science Computer Review35 (2): 167–83.
43.
NadeauRichardNiemiRichard G.1995. “Educated Guesses: The Process of Answering Factual Knowledge Questions in Surveys.”Public Opinion Quarterly59 (3): 323.
44.
PriorMarkusLupiaArthur. 2008. “Money, Time, and Political Knowledge: Distinguishing Quick Recall and Political Learning Skills.” American Journal of Political Science52 (1): 169–83.
45.
PriorMarkus. 2010. “You’ve Either Got It or You Don’t? The Stability of Political Interest over the Life Cycle.”The Journal of Politics72 (3): 747–66.
46.
ShulmanHillary C.BosterFranklin J.2014. “Effect of Test-Taking Venue and Response Format on Political Knowledge Tests.” Communication Methods and Measures8 (3): 177–89.
47.
StrabacZ.AalbergT.2011. “Measuring Political Knowledge in Telephone and Web Surveys: A Cross-National Comparison.”Social Science Computer Review29 (2): 175–92.
48.
VezzoniCristianoLadiniRiccardo. 2017. “Thou Shalt Not Cheat: How to Reduce Internet Use in Web Surveys on Political Knowledge.” Italian Political Science Review/Rivista Italiana Di Scienza Politica47 (3): 251–65.
49.
WebsterDonna M.KruglanskiArie W.1994. “Individual Differences in Need for Cognitive Closure.” Journal of Personality and Social Psychology67 (6): 1049–62.
50.
ZallerJohn R.1992. The Nature and Origins of Mass Opinion. Cambridge: Cambridge University Press.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
