Does the opportunity to deliver visual instead of verbal stimuli of political knowledge to respondents in web-based surveys make a difference in terms of data quality? For instance, does the presentation of visual knowledge items reduce cheating, that is, looking up the answer via the Web? And do visual and verbal stimuli capture the same underlying construct? To test whether the use of visuals to measure political knowledge effectively makes a difference, we administer a question form experiment in an online survey of the German Longitudinal Election Study. Respondents are randomly assigned to one of two question formats—visual or verbal—and are asked to solve a set of eight questions on political leaders and their offices. The instruments are validated based on nonparametric item response theory and analyses of response latency. While there is no clear evidence for cheating behavior under either of the conditions, both instruments form strong knowledge scales. Results from a regression analysis indicate that the scales provide measures of closely related but not identical concepts.
AlvarezR. M. (1998). Information and Elections. Revised to Include the 1996 Presidential Election. Ann Arbor: University of Michigan Press.
2.
AnsolabehereS.SchafferB. F. (2014). Does survey mode still matter? Findings from a 2010 multi-mode comparison. Political Analysis, 22, 285–303.
3.
BarabasJ. (2002). Another look at the measurement of political knowledge. Political Analysis, 10, 209–223.
4.
BartelsL. M. (1996). Uninformed votes: Information effects in presidential elections. American Journal of Political Science, 40, 194–230.
5.
BenzM.StutzerA. (2004). Are voters better informed when they have a larger say in politics?—Evidence for the European Union and Switzerland. Public Choice, 119, 31–59.
6.
BernsteinR.ChadhaA.MontjoyR. (2001). Overreporting voting: Why it happens and why it matters. Public Opinion Quarterly, 65, 22–44.
7.
Booth-KewleyS.LarsonG. E.MiyoshiD. K. (2007). Social desirability effects on computerized and paper-and-pencil questionnaires. Computers in Human Behavior, 23, 463–477.
8.
BrownJ. D. (2012). Understanding the better than average effect: Motives (still) matter. Personality and Social Psychology Bulletin, 38, 209–219.
9.
CallegaroM.YangY.BholaD. S.DillmanD. A.ChinT.-Y. (2009). Response latency as an indicator of optimizing in online questionnaires. Bulletin of Sociological Methodology, 103, 5–25.
10.
CliffordS.JeritJ. (2014). Is there a cost to convenience? An experimental comparison of data quality in laboratory and online studies. Journal of Experimental Political Science, 1, 120–131.
11.
CliffordS.JeritJ. (2015). Cheating on knowledge questions in online surveys: An assessment of the problem and solutions(Unpublished Manuscript).
12.
CouperM. P.BosnjakM. (2010). Internet surveys. In MarsdenP.WrightJ. (Eds.), Handbook of survey research (2nd ed., pp. 527–550). Bingley, England: Emerald Group Publishing Limited.
13.
CouperM. P.TourangeauR.KenyonK. (2004). Picture this! Exploring visual effects in web surveys. Public Opinion Quarterly, 68, 255–266.
14.
DeBellM. (2013). Harder than it looks: Coding political knowledge on the ANES. Political Analysis, 21, 393–406.
15.
Delli CarpiniM. X.KeeterS. (1993). Measuring political knowledge: Putting first things first. American Journal of Political Science, 37, 1179–1206.
16.
Delli CarpiniM. X.KeeterS. (1996). What Americans know about politics and why it matters. New Haven, CT: Yale University Press.
17.
EloK. (2009). Asking factual knowledge questions. Reliability in web-based, passive sampling surveys. Social Science Computer Review, 27, 1–16.
18.
FerejohnJ. A.KuklinskiJ. H. (1990). Information and democratic processes. Chicago: University of Illinois Press.
19.
GordonS. B.SeguraG. M. (1997). Cross-national variation in the political sophistication of individuals: Capability or choice?Journal of Politics, 59, 126–147.
20.
GraberD. A. (1990). Seeing I remembering. How visuals contribute to learning from television news. Journal of Communication, 40, 134–155.
21.
GraberD. A. (2001). Processing politics: Learning from television in the internet age. Chicago: University of Chicago Press.
22.
GrüterT.GrüterM.CarbonC.-C. (2008). Neural and genetic foundations of face recognition and prosopagnosia. Journal of Neuropsychology, 2, 79–97.
JensenC.ThomsenJ. P. F. (2014). Self-reported cheating in web surveys on political knowledge. Quality & Quantity, 48, 3343–3354.
25.
KreuterF.CouperM. P.LybergL. (2010). The use of paradata to monitor and manage survey data collection. In Proceedings of the American Statistical Association’s Survey Research Methods Section (pp. 282–296). Alexandria, VA: American Statistical Association.
26.
KrosnickJ. A.AlwinD. F. (1987). Satisficing: A strategy for dealing with the demands of survey questions. Columbus: Ohio State University.
27.
LeighleyJ. (1991). Participation as a stimulus of political conceptualization. Journal of Politics, 53, 198–211.
LuskinR. C. (1987). Measuring political sophistication. American Journal of Political Science, 31, 856–899.
30.
LuskinR. C. (1990). Explaining political sophistication. Political Behavior, 12, 331–361.
31.
LuskinR. C. (2002). From denial to extenuation (and finally beyond): Political sophistication and citizen performance. In KuklinskiJ. H. (Ed.), Thinking about political psychology (pp. 281–301). Cambridge, England: Cambridge University Press.
32.
MacdonaldS. E.RabinowitzG.ListhaugO. (1995). Political sophistication and models of issue voting. British Journal of Political Science, 25, 453–483.
33.
McCandlissB. D.NobleK. G. (2003). The development of reading impairment: a cognitive neuroscience model. Mental retardation and developmental disabilities research reviews, 9, 196–205.
34.
MillarM.DillmanD. A. (2012). Do mail and internet surveys produce different item nonresponse rates? An experiment using random mode assignment. Survey Practice, 5, 1–6.
35.
MokkenR. J. (1971). A theory and procedure of scale analysis. Berlin, Germany: De Gruyter.
36.
MondakJ. J. (2000). Reconsidering the measurement of political knowledge. Political Analysis, 8, 57–82.
37.
MondakJ. J. (2001). Developing valid knowledge scales. American Journal of Political Science, 45, 224–238.
38.
PriorM. (2014). Visual political knowledge: A different road to competence?The Journal of Politics, 76, 41–57.
39.
PriorM.LupiaA. (2008). Money, time, and political knowledge: Distinguishing quick recall and political learning skills. American Journal of Political Science, 52, 169–183.
40.
RattingerH.RoßteutscherS.Schmitt-BeckR.WeßelsB.WolfC. (2014). Long-term online tracking, T23 (GLES). ZA5723 Data file Version 1.0.0, doi:10.4232/1.11880. Cologne, Germany: GESIS Data Archive.
41.
SelbP.LachatR. (2009). The more, the better? Counterfactual evidence on the effect of compulsory voting on the consistency of party choice. European Journal of Political Research, 48, 573–597.
42.
ShulmanH. C.BosterF. J. (2014). Effect of test-taking venue and response format on political knowledge tests. Communication Methods and Measures, 8, 177–189.
43.
SijtsmaK.MolenaarI. W. (2002). Introduction to nonparametric item response theory. Thousand Oaks, CA: Sage.
44.
SnidermanP. M.BrodyR. A.TetlockP. E. (1993). Reasoning and choice. Explorations in political psychology. Cambridge, England: Cambridge University Press.
StrabacZ.AalbergT. (2011). Measuring political knowledge in telephone and web surveys: A cross-national comparison. Social Science Computer Review, 29, 175–192.
47.
SturgisP.AllumN.SmithP. (2008). An experiment on the measurement of political knowledge in surveys. Public Opinion Quarterly, 85, 90–102.
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
