Perceptions of opinion poll credibility: The role of partisan bias

Abstract

What determines the credibility of opinion polls, and for whom? Opinion polls assessing voting intention are an increasingly prominent aspect of pre-election media discourse, yet they often vary in terms of their estimations, who conducted the poll, and where the poll is ultimately published. This research note considers the determinants of pre-election poll credibility when citizens are exposed to conflicting polls, contributing to a growing field which considers the subjectivity with which partisan-relevant information is perceived in polarised contexts. Using a conjoint analysis in Turkey, this paper produces strong evidence that polls are perceived differently across partisan lines. Whereas government supporters rely almost exclusively on a poll’s result, opposition supporters place considerably more weight on the media source when determining poll credibility.

Pre-election opinion polls are an increasingly prominent aspect of election coverage. Prior to many democratic contests around the world, polls are frequently published estimating the support for candidates and parties throughout the campaign. Such polls are used to gauge government performance, to direct policy or campaign strategies, and may even inadvertently affect voting behaviour (Jacobs and Shapiro, 2005; Lau and Redlawsk, 2006; Simon, 1954). However, polls often vary in many respects and, consequently, so too does their perceived credibility. Furthermore, prospective voters are likely to encounter several polls prior to the election that may differ in several respects. It therefore falls upon the individual to determine which pre-election polls to give more credence to than others.

In recent years, a considerable body of research has established the subjectivity with which opinion polling information is often perceived. Contrary to polling practitioners or scholars of public opinion, the general public on average do not prioritise concerns of representative samples, question design, or other aspects of survey methodology (Panagopoulos et al., 2009). Rather, most people are likely to attribute credibility to polls that are in some way congruent with their beliefs or preferences, which often relates to an identity group such as a political party (Kuru et al., 2017). Yet despite these advances, we still do not fully understand exactly what the determinants of polling credibility are, and whether this may vary for different partisan groups, for example. Opinion polls can and often do vary significantly in terms of the estimations they report, where and how this information is disseminated, and who conducted the polls in the first place.

Using a conjoint analysis in the highly polarised and competitive context of Turkish municipal elections, this research note finds support for the notion that the credibility of polls is determined subjectively and advances our understanding of the ways in which polls are viewed differently across partisan groups. While there is little evidence that people assign credibility based on who conducted the poll, government supporters’ give more credence to a poll based almost exclusively on its content (such as who is winning and by how much), whereas opposition supporters consider the media source to a much greater extent.

The determinants of opinion poll credibility

Opinion polls are not perceived equally by all. What determines a poll’s credibility – or how trustworthy or believable the poll is perceived to be – has been the subject of significant research in recent years. Three features of opinion polls can be distinguished that previous work has identified as the determining factors of their credibility. The first of these concerns the methodology used to conduct the poll, or the reputation of the polling vendor itself (Crespi, 1988; Panagopoulos et al., 2009). For experts and practitioners, what constitutes a credible poll is typically determined by the size and representativeness of the sample, the validity and precision of question wording, and/or the way in which results are related to the audience. From this perspective, what constitutes a credible poll is relatively clear and scientifically objective. A poll on an upcoming national election conducted online using convenience sampling would undeniably be less credible than a face-to-face survey using a nationally representative sample, for example. However, many in the general public are likely to be far less attentive to such factors when determining poll credibility (Panagopoulos et al., 2009). While significant differences in poll quality – such as that above – are indeed considered by the public (Kuru et al., 2020), distinctions made on this basis are less likely when the difference in quality is marginal.

Rather, two further features of opinion polls have been shown to be far more influential for the public. A growing field of research has consistently shown opinion polls to be evaluated more favourably when their results are in congruence with the beliefs or preferences of the respondent (Kuru et al., 2017; Madson and Hillygus, 2020). In other words, the perceived credibility of opinion polls is subjectively determined by its content. This process can take one of two forms, as individuals may consider information in relation to accuracy-seeking or directional goals (Kahan, 2016; Kunda, 1990; Taber and Lodge, 2006). In other words, opinion polls may be viewed as more credible if they are perceived to be closer to the truth or if they are congruent with an individual’s preferences. Directional motivations are thought to be of particular significance in partisan-relevant issues such as opinion polls (Jerit and Barabas, 2012).

Relatedly, the third feature that has been found to be of particular relevance is the media source. Information is generally preferred if it originates from credible sources (Druckman, 2001). The election coverage of partisan media has been shown to affect the electoral expectations of voters (Searles et al., 2018), suggesting an increased trust in information disseminated by like-minded media channels. That being said, some research questions the effect of ideologically congruent media when the information is counter-attitudinal (Kuru et al., 2017), suggesting a possible interdependency between content and source.

Taken together, these three features of polls constitute the bulk of the proposed explanations of their credibility. That being said, there remain several aspects left unexplored by these studies. While much previous work has been experimental, a frequent assumption taken by these designs is that polls are singular and/or coherent. Much of this work typically makes comparison between experimental groups, each of whom have only been exposed to a single poll. One notable exception to this is the work of Kuru et al. (2020), but their focus is rather on the poll quality, which this study identifies as distinct from credibility. Secondly, while this field has established that polls can be perceived in different ways by different groups of people, it does not fully explore whether the relationships are heterogenous across identity groups such as political partisans. In other words, while it has been established that certain identity groups (such as partisans) perceive polls differently, it is not clear whether the same attributes of this information matter to each group to the same degree. Both of these aspects are addressed by the present study.

Research design

The subjective determinants of opinion poll credibility are assessed in a pre-registered conjoint experiment conducted in Turkey in 2020. ¹ Conjoint analysis is well-suited in this case as it enables the quantification and comparison of the independent effects of several factors in a single test (Hainmueller et al., 2014). If credibility is indeed attributed subjectively by individuals, this approach is ideal in being able to tease out which factors matter more or less than others. Turkish mayoral elections are a useful case in this regard, due to the high levels of political competition at the metropolitan municipal level, a polarised media environment, and a history of considerable variations in the estimations of pollsters around elections. While the independence and freedom of media institutions in Turkey is more restricted than in more democratic contexts, similar patterns of polarisation are identifiable in many elections in states with a comparably polarised electorate and media landscape.

In the experiment, subjects were presented with two polls that referred to a hypothetical upcoming municipal election. ² The polls were randomised along three dimensions that were also presented to the respondent: the content (which party is leading and by how much), the media source (newspaper), and the polling vendor. In addition to this information, the polls were also presented graphically as a bar chart illustrating the position of the five largest parties. ³ See Figure 1 for an example.

Figure 1.

Conjoint analysis poll set example.

After each set of two polls, respondents were asked to identify which they would be more likely to take into consideration. The ambiguity with regard to the question wording is intentional and reflects the aim of the set-up not to trigger, for example, accuracy or directional motivations and consequently a specific attribute of the polls. This may well be the case if the approach is taken to measure credibility based on the perceived reliability, trustworthiness, or accuracy of sources (Kuru et al., 2017; Moreno et al., 2014). For example, asking respondents to identify to the most accurate poll may trigger accuracy motivations and lead to the comparison of experimental treatments to the state of public opinion outside of experimental conditions. Specifically referring to the quality or reliability of a poll is also likely to implicitly encourage respondents to focus more on the polling vendor or media source, respectively. The ambiguous conception of credibility measured here is therefore closer to believability than perceived accuracy or quality. While the respondents may equate these concepts this study does not assume as much.

806 respondents were presented with six sets of polls to choose between consecutively, resulting in the evaluation of a total of 4836 poll sets and 9672 polls overall. Quota-based representative sampling was limited to the metropolitan municipalities of Turkey, where first past the post electoral rules are used elect a single mayor, and elections are generally much more competitive relative to the dominant position the incumbent Justice and Development Party (AKP) holds in rural areas. ⁴ Quotas based on age and gender were used to mitigate the imbalance of online panels based on factors that are likely to correlate with party preference. Descriptive statistics for the balance of demographics across partisan groups can be found in Appendices 1 and 2. The conjoint analysis was embedded in an online survey that was conducted by Dynata in June 2020.

The randomised poll dimensions – or attributes – could take one of several pre-determined levels. The lead attribute can take the value of a four- or nine-point lead for the AKP (national incumbent) or Republican People’s Party (CHP, the major opposition party) or a dead heat, and are selected on the basis that they could be viewed as a relatively realistic reflection of the current state of political competition in local Turkish politics in the largest cities. ^{5, 6} Significant variations of this in local polling estimates are relatively frequent in Turkey. For example, ahead of the 2019 local elections in the city of Adana, a Gezici poll indicated a 9-point lead for the incumbent People’s Alliance (MHP and AKP). The eventual result just 2 weeks later was a 11-point victory for the opposition Nation Alliance. At the same time in Bursa, a Metropoll survey 3 weeks prior to the contest indicated a 7-point Nation Alliance lead in a contest that was eventually won by the People’s Alliance with a 3-point majority.

The attribute levels of the polling company and newspaper are selected on the basis that they represent a range of presumed biases, not only in terms of the left-right political-economic scale, but also in terms of whether they are supportive or critical of the Erdoğan regime. The newspapers included in the analysis consist of (crudely, from left to right) Sözcü, Cumhuriyet, Posta, Hürriyet, Sabah and Yeni Şafak. The polling companies consist of IPSOS (international), and Konda, Gezici, GENAR, and Metropoll (domestic). Each polling vendor is a professional organisation, yet several have been accused of political biases in recent years, which allows subjective perceptions of relative quality to vary despite the polls being relatively (objectively) high quality. The domestic pollsters included range between the neutral and those with alleged bias in favour of either the AKP or CHP, due to a history of over- or under-estimations of a particular party, or personal linkages. Konda, for example, had been criticised for high polling results favouring AKP leader Erdoğan prior to the presidential election in 2014. The company later apologised for their inaccurate estimations (Nediyor, 2014). Such a media landscape is largely representative of the ideological range of the mainstream print media in Turkey, and reflects the partisan polarisation of Turkish society (Laebens and Öztürk, 2020).

Results

The analysis uses marginal means for each attribute level obtained from a linear regression model in which the likelihood of a poll being selected is the function of all poll attribute levels as dummy variables. As there were six repetitions for each respondent, standard errors in the analysis were clustered at the respondent level. The marginal mean corresponds to the likelihood of selecting a given poll when an attribute (e.g., lead) is a specific level (e.g., AKP +4). ⁷ Available in Appendices 3 and 4, regression analyses were run on a pooled sample as well as subsamples of government and opposition supporters, which are illustrated in Figure 2. Respondents are coded as government supporters if their stated preferred party is either the AKP or MHP (who together form the People’s Alliance) and as opposition supporters if their preferred party is either the CHP or İYİ. ⁸ Of the 806 respondents, 277 (34%) were coded as government supporters, 367 (46%) were opposition supporters and the remaining 162 were coded as non-partisan as they either did not state a preferred party, or their preferred party was not part of one of the two major electoral alliances.

Figure 2.

The effect of poll attributes on perceived credibility.

The results in Figure 2 clearly illustrate heterogeneous determinants of opinion poll credibility across partisan lines. Government supporters are much more likely to place their trust in polls that show the AKP to be leading and are significantly more sceptical of those that show a lead for the rival party. ⁹ This is overwhelmingly the strongest determinant of opinion poll credibility for government supporters. ¹⁰ While there is also a preference for polls beneficial to the in-party for opposition supporters (polls indicating a CHP lead), this relationship seems not to be linear as with government-supporters, as a nine-point opposition lead is not viewed as any more credible than a four-point lead. This result leans more toward a demonstration of accuracy-seeking motivations, or perhaps directional motivations to a point; polls illustrating a large lead may be viewed as too good to be true, at least in relation to a more moderate lead.

Results are presented for the whole sample (left, n=806), government-supporters (middle, n=277) and opposition-supporters (right, n=367). Respondents were coded as government supporters if their preferred party is the AKP or MHP (Nationalist Movement Party), and opposition supporters if they preferred the CHP or İYİ (Good Party), which reflects cross-party alliances in the 2019 local elections. ¹¹ Horizontal bars represent 95% confidence intervals for marginal means. For marginal means see Appendices 3 and 4.

While the pollster is similarly uninfluential for both government and opposition supporters, there is divergence in how these groups view the source of this information. For opposition supporters, there is considerably more weight placed on this factor than in the case of government supporters. Above all else, opposition-supporters favour and deem more credible polls that are published in Cumhuriyet and Sözcü – two of the most well-known government-critical newspapers. For government supporters, the media source matters considerably less, provided that the information is beneficial to the in-party.

Discussion and conclusion

The results presented here not only provide additional support for the notion that opinion poll credibility is subjectively determined, but also illustrate the heterogeneity with which credibility is attributed between partisan groups. While perceptions of credibility are equally biased across partisan lines, this bias tends toward different aspects of polling information.

Government supporters seemingly interpret a poll’s credibility almost exclusively according to how it positions their favoured party. As far as the attribute levels of the analysis allow us to infer, this relationship also seems linear; bigger leads for the in-party are more credible whereas bigger leads for the out-party are less credible. At least as far as government supporters are concerned, pre-election polls do indeed seem to invoke a motivated reasoning or – at the very least –wishful thinking (Madson and Hillygus, 2020; Searles et al., 2018). There is evidence to suggest that this is also the case (albeit to a lesser degree) for opposition supporters. However, that a nine-point opposition lead is not viewed as any more or less credible than a four-point deficit suggests that opposition voters may have a somewhat more modest view of their electoral prospects. In contrast to the repeated reinforcement of negative information (e.g., Redlawsk et al., 2010), this hints at the existence of a kind of motivated tipping point beyond which preferential information seems too good to be true, encouraging individuals begin to engage in accuracy-seeking behaviour.

While the content of polls also matters for opposition supporters, they are also much more attentive to the source of this information and their ideological biases. In this case, opposition supporters give significantly less credence to polls published in newspapers affiliated with the incumbent party, rather favouring polls published in newspapers that are critical of the government. Given that there is little such concern for government supporters, this is the most significant difference between the two groups. Future research efforts should therefore try to determine to what extent greater attentiveness to where polls are published is a feature of purely being in opposition in a polarised media environment, or whether this phenomenon is amplified when the independence of the media is under threat to the extent that it is in Turkey. The mistrust of rival media organisations and the political information they disseminate is not limited to such contexts, however, and the biases in how this information is processed are likely to be present even in more nominally democratic contests with similarly polarised political landscapes.

Supplemental Material

Supplemental Material - Perceptions of opinion poll credibility: The role of partisan bias

Supplementary Material for Perceptions of opinion poll credibility: The role of partisan bias by Stephen Dawson in Party Politics.