Sage Journals: Discover world-class research

Abstract

Over the last 40 years, affective polarization has noticeably increased, with recent elections showing especially large gains. Moreover, these increases have surpassed other indicators of polarization, such as issue attitudes and partisan-ideological sorting. Could the recent surge in affective polarization be influenced by the growing use of internet-based, self-administered surveys? I argue that measures of partisan affect are more vulnerable to mode effects than other indicators of polarization. In this paper, I employ American National Election Studies panel data from 2016 to 2020. Crucially, in 2016 the survey was conducted with a dual mode in which a portion of respondents were interviewed in the traditional face-to-face format while others completed a self-administered online survey. Both sets of respondents were re-interviewed in 2020 where all respondents completed the survey online. I find that differences in party affective polarization and candidate trait ratings between the two modes, substantial in 2016, largely disappear when social desirability effects are removed. In contrast, issue extremity and partisan-ideological sorting show no noticeable mode effects.

Keywords

Survey mode effects affective polarization issue extremity sorting

Get full access to this article

View all access options for this article.

References

Abramowitz

A. I.

Saunders

K. L.

(2008). Is polarization a myth? The Journal of Politics, 70(2), 542–556. https://doi.org/10.1017/s0022381608080493

Abramowitz

A. I.

Webster

S. W.

(2018). Negative partisanship: Why Americans dislike parties but behave like rabid partisans. Political Psychology, 39(S1), 119–135. https://doi.org/10.1111/pops.12479

Aichholzer

(2013). Intra-individual variation of extreme response style in mixed-mode panel studies. Social Science Research, 42(3), 957–970. https://doi.org/10.1016/j.ssresearch.2013.01.002

Ansolabehere

Schaffner

B. F.

(2015). Distractions: The incidence and consequences of interruptions for survey respondents. Journal of Survey Statistics and Methodology, 3(2), 216–239. https://doi.org/10.1093/jssam/smv003

Bafumi

Shapiro

R. Y.

(2009). A new partisan voter. The Journal of Politics, 71(1), 1–24. https://doi.org/10.1017/s0022381608090014

Bartels

L. M.

(2000). Partisanship and voting behavior, 1952-1996. American Journal of Political Science, 44(1), 35. https://doi.org/10.2307/2669291

Berinsky

A. J.

(2004). Can we talk? Self-presentation and the survey response. Political Psychology, 25(4), 643–659. https://doi.org/10.1111/j.1467-9221.2004.00391.x

Blackwelder

W. C.

(1982). Proving the null hypothesis’ in clinical trials. Controlled Clinical Trials, 3(4), 345–353. https://doi.org/10.1016/0197-2456(82)90024-1

Bougher

L. D.

(2017). The correlates of discord: Identity, issue alignment, and political hostility in polarized America. Political Behavior, 39(3), 731–762. https://doi.org/10.1007/s11109-016-9377-1

10.

Bowyer

B. T.

Rogowski

J. C.

(2017). Mode <atters: Evaluating response comparability in a mixed-mode survey. Political Science Research and Methods, 5(2), 295–313. https://doi.org/10.1017/psrm.2015.28

11.

Campbell

Converse

P. E.

Miller

W. E.

Stokes

D. E.

(1960). The American voter: Wiley.

12.

Cavari

A. G.

Freedman

(2018). Polarized mass or polarized few? Assessing the parallel rise of survey nonresponse and measures of polarization. The Journal of Politics, 80(2), 719–725. https://doi.org/10.1086/695853

13.

Chang

Krosnick

J. A.

(2009). National surveys via RDD telephone interviewing versus the internet: Comparing sample representativeness and response quality. Public Opinion Quarterly, 73(4), 641–678. https://doi.org/10.1093/poq/nfp075

14.

Chang

Krosnick

J. A.

(2010). Comparing oral interviewing with self-administered computerized questionnaires: An experiment. Public Opinion Quarterly, 74(1), 154–167. https://doi.org/10.1093/poq/nfp090

15.

Christenson

D. P.

Weisberg

H. F.

(2019). Bad characters or just more polarization? The rise of extremely negative feelings for presidential candidates. Electoral Studies, 61, Article 102032. https://doi.org/10.1016/j.electstud.2019.03.008

16.

Connors

E. C.

(2023). Social desirability and affective polarization. Public Opinion Quarterly.

17.

Deming

W. E.

(1944). On errors in surveys. American Sociological Review, 9(4), 359–369. https://doi.org/10.2307/2085979

18.

Dias

Lelkes

(2022). The nature of affective polarization: Disentangling policy disagreement from partisan identity. American Journal of Political Science, 66(3), 775–790. https://doi.org/10.1111/ajps.12628

19.

Dunning

(2008). Improving causal inference: Strengths and limitations of natural experiments. Political Research Quarterly, 61(2), 282–293. https://doi.org/10.1177/1065912907306470

20.

Enders

A. M.

(2021). Issues versus affect: How do elite and mass polarization compare? The Journal of Politics, 83(4), 1872–1877. https://doi.org/10.1086/715059

21.

Finkel

E. J.

Bail

C. A.

Cikara

Ditto

P. H.

Iyengar

Klar

Mason

McGrath

M. C.

Nyhan

Rand

D. G.

Skitka

L. J.

Tucker

J. A.

Van Bavel

J. J.

Wang

C. S.

Druckman

J. N.

(2020). Political sectarianism in America. Science, 370(6516), 533–536. https://doi.org/10.1126/science.abe1715

22.

Fiorina

M. P.

Abrams

S. J.

Pope

J. C.

(2008). Polarization in the American public: Misconceptions and misreadings. The Journal of Politics, 70(2), 556–560. https://doi.org/10.1017/s002238160808050x

23.

Fisher

R. J.

(1993). Social desirability bias and the validity of indirect questioning. Journal of Consumer Research, 20(2), 303–315. https://doi.org/10.1086/209351

24.

Frankel

L. L.

Hillygus

D. S.

(2014). Looking beyond demographics: Panel attrition in the ANES and GSS. Political Analysis, 22(3), 336–353. https://doi.org/10.1093/pan/mpt020

25.

Fricker

Galesic

Tourangeau

Yan

(2005). An experimental comparison of web and telephone surveys. Public Opinion Quarterly, 69(3), 370–392. https://doi.org/10.1093/poq/nfi027

26.

Gill

(2022). Testing the effect of cross-cutting exposure to cable TV news on affective polarization: Evidence from the 2020 US presidential election. Journal of Broadcasting & Electronic Media, 66(2), 320–339. https://doi.org/10.1080/08838151.2022.2087653

27.

Goldenbeld

De Craen

(2013). The comparison of road safety survey answers between web-panel and face-to-face: Dutch results of SARTRE-4 survey. Journal of Safety Research, 46, 13–20. https://doi.org/10.1016/j.jsr.2013.03.004

28.

Gooch

Vavreck

(2019). How face-to-face interviews and cognitive skill affect item non-response: A randomized experiment assigning mode of interview. Political Science Research and Methods, 7(1), 143–162. https://doi.org/10.1017/psrm.2016.20

29.

Green

D. P.

Palmquist

(1994). How stable is party identification? Political Behavior, 16(4), 437–466. https://doi.org/10.1007/bf01498826

30.

Green

D. P.

Platzman

(2022). Partisan stability during turbulent times: Evidence from three American panel surveys. Political Behavior, 46(1), 377–403. https://doi.org/10.1007/s11109-022-09825-y

31.

Halliez

A. A.

Thornton

J. R.

(2021). Examining trends in ideological identification: 1972–2016. American Politics Research, 49(3), 259–268. https://doi.org/10.1177/1532673x20961314

32.

Hanson

R. H.

Marks

E. S.

(1958). Influence of the interviewer on the accuracy of survey results. Journal of the American Statistical Association, 53(283), 635–655. https://doi.org/10.2307/2282043

33.

Heerwegh

(2009). Mode differences between face-to-face and web surveys: An experimental investigation of data quality and social desirability effects. International Journal of Public Opinion Research, 21(1), 111–121. https://doi.org/10.1093/ijpor/edn054

34.

Heerwegh

Loosveldt

(2008). Face-to-Face versus web surveying in a high-internet-coverage population: Differences in response quality. Public Opinion Quarterly, 72(5), 836–846. https://doi.org/10.1093/poq/nfn045

35.

Hetherington

M. J.

(2001). Resurgent mass partisanship: The role of elite polarization. American Political Science Review, 95(3), 619–631. https://doi.org/10.1017/s0003055401003045

36.

Hill

Tausanovitch

(2015). A disconnect in representation? Comparison of trends in congressional and public polarization. The Journal of Politics, 77(4), 1058–1075. https://doi.org/10.1086/682398

37.

Hobolt

S. B.

Lawall

Tilley

(2023). The polarizing effect of partisan echo chambers. American Political Science Review. https://doi.org/10.1017/S0003055423001211

38.

Hochstim

J. R.

(1967). A critical comparison of three strategies of collecting data from households. Journal of the American Statistical Association, 62(319), 976–989. https://doi.org/10.2307/2283686

39.

Iyengar

Krupenkin

(2018). The strengthening of partisan affect. Political Psychology, 39(S1), 201–218. https://doi.org/10.1111/pops.12487

40.

Iyengar

Lelkes

Levendusky

Malhotra

Westwood

S. J.

(2019). The origins and consequences of affective polarization in the United States. Annual Review of Political Science, 22(1), 129–146. https://doi.org/10.1146/annurev-polisci-051117-073034

41.

Iyengar

Westwood

(2015). Fear and loathing across party lines: New evidence on group polarization. American Journal of Political Science, 59(3), 690–707. https://doi.org/10.1111/ajps.12152

42.

Jackman

Spahn

(2019). Why does the American national election study overestimate voter turnout? Political Analysis, 27(2), 193–207. https://doi.org/10.1017/pan.2018.36

43.

Kalmoe

N. P.

(2020). Uses and abuses of ideology in political psychology. Political Psychology, 41(4), 771–793. https://doi.org/10.1111/pops.12650

44.

Karp

J. A.

Brockington

(2005). Social desirability and response validity: A comparative analysis of overreporting voter turnout in five countries. The Journal of Politics, 67(3), 825–840. https://doi.org/10.1111/j.1468-2508.2005.00341.x

45.

Keith

B. E.

Magleby

D. B.

Nelson

C. J.

Orr

Westlye

M. C.

Wolfinger

R. E.

(1992). The myth of the independent voter: University of California Press.

46.

Kingzette

Druckman

J. N.

Klar

Krupnikov

Levendusky

Ryan

J. B.

(2021). How affective polarization undermines support for democratic norms. Public Opinion Quarterly, 85(2), 663–677. https://doi.org/10.1093/poq/nfab029

47.

Klar

Krupnikov

(2016). Independent politics: How American disdain for parties leads to political inaction. Cambridge University Press.

48.

Kreuter

Presser

Tourangeau

(2008). Social desirability bias in CATI, IVR, and web surveys: The effects of mode and question sensitivity. Public Opinion Quarterly, 72(5), 847–865. https://doi.org/10.1093/poq/nfn063

49.

Krysan

(1998). Privacy and theExpression of white racial attitudes: A comparison across three contexts. Public Opinion Quarterly, 62(4), 506–544. https://doi.org/10.1086/297859

50.

Lelkes

(2016). Mass polarization: Manifestations and measurements. Public Opinion Quarterly, 80(S1), 392–410. https://doi.org/10.1093/poq/nfw005

51.

Levendusky

M. S.

(2009). The partisan sort: How liberals became democrats and conservatives became Republicans: University of Chicago Press.

52.

Liu

Wang

(2015). Data collection mode effect on feeling thermometer questions: A comparison of face-to-face and web surveys. Computers in Human Behavior, 48(July), 212–218. https://doi.org/10.1016/j.chb.2015.01.057

53.

Locander

Sudman

Bradburn

(1976). An investigation of interview method, threat and response distortion. Journal of the American Statistical Association, 71(354), 269–275. https://doi.org/10.1080/01621459.1976.10480332

54.

Magleby

D. B.

Nelson

C. J.

Westlye

M. C.

(2011). The myth of the independent voter revisited. In Sniderman

P. M.

Highton

(Eds.), Facing the challenge of democracy: Explorations in the analysis of public opinion and political participation (pp. 238–264): Princeton University Press.

55.

Mason

(2015). ‘I disrespectfully agree’: The differential effects of partisan sorting on social and issue polarization. American Journal of Political Science, 59(1), 128–145. https://doi.org/10.1111/ajps.12089

56.

Mason

(2016). A cross-cutting calm: How social sorting drives affective polarization. Public Opinion Quarterly, 80(S1), 351–377. https://doi.org/10.1093/poq/nfw001

57.

Mason

(2018). Ideologues without issues: The polarizing consequences of ideological identities. Public Opinion Quarterly, 82(S1), 280–301. https://doi.org/10.1093/poq/nfy005

58.

Mellon

Prosser

(2021). Correlation with time explains the relationship between survey nonresponse and mass polarization. The Journal of Politics, 83(1), 390–395. https://doi.org/10.1086/709433

59.

Menchaca

(2023). Are Americans polarized on issue dimensions? Journal of Elections, Public Opinion, and Parties, 33(2), 228–246. https://doi.org/10.1080/17457289.2021.1910954

60.

Millham

Kellogg

R. W.

(1980). Need for social approval: Impression management or self-deception? Journal of Research in Personality, 14(4), 445–457. https://doi.org/10.1016/0092-6566(80)90003-3

61.

Orr

L. V.

Huber

G. A.

(2020). The policy basis of measured partisan animosity in the United States. American Journal of Political Science, 64(3), 569–586. https://doi.org/10.1111/ajps.12498

62.

Paulhus

D. L.

(1984). Two-component models of socially desirable responding. Journal of Personality and Social Psychology, 46(3), 598–609. https://doi.org/10.1037/0022-3514.46.3.598

63.

Petrocik

J. R.

(2009). Measuring party support: Leaners are not independents. Electoral Studies, 28(4), 562–572. https://doi.org/10.1016/j.electstud.2009.05.022

64.

Phillips

J. B.

(2024). Affective polarization and habits of political participation. Electoral Studies, 87, Article 102733. https://doi.org/10.1016/j.electstud.2023.102733

65.

Rainey

(2014). Arguing for a negligible effect. American Journal of Political Science, 58(4), 1083–1091. https://doi.org/10.1111/ajps.12102

66.

Saunders

K. L.

Abramowitz

A. I.

(2004). Ideological realignment and active partisans in the American electorate. American Politics Research, 32(3), 285–309. https://doi.org/10.1177/1532673x03259195

67.

Tourangeau

Rips

L. J.

Rasinski

(2000). The psychology of survey response. Cambridge University Press.

68.

Tyler

Shanto

(2024). Testing the robustness of the ANES feeling thermometer indicators of affective polarization. American Political Science Review, 118(3), 1570–1576. https://doi.org/10.1017/s0003055423001302

69.

Vannieuwenhuyze

J. T. A.

Loosveldt

Molenberghs

(2012). A method to evaluate mode effects on the mean and variance of a continuous variable in mixed-mode surveys. International Statistical Review, 80(2), 306–322. https://doi.org/10.1111/j.1751-5823.2011.00167.x

70.

Villa

J. M.

(2016). diff: Simplifying the estimation of difference-in-differences treatment effects. STATA Journal: Promoting communications on statistics and Stata, 16(1), 52–71. https://doi.org/10.1177/1536867x1601600108

71.

Webster

S. W.

Abramowitz

A. I.

(2017). The ideological foundations of affective polarization in the U.S. Electorate. American Politics Research, 45(4), 621–647. https://doi.org/10.1177/1532673x17703132

72.

Westlake

W. J.

(1979). Statistical aspects of comparative bioavailability trials. Biometrics, 35(1), 273–280.

73.

Wiseman

(1972). Methodological bias in public opinion surveys. Public Opinion Quarterly, 36(1), 105–108. https://doi.org/10.1086/267981

74.

Zingher

J. N.

(2022). Political choice in a polarized America: How elite polarization shapes mass behavior. Oxford University Press.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.30 MB

The Impact of Survey Mode on the Measurement of Polarization in the Public: 2016–2020 U.S. Panel Evidence

Abstract

Keywords

Get full access to this article

References

Supplementary Material