Abstract
The foundational assumptions of policy diffusion relate to the behavior of individual policymakers. However, empirical tests of policy diffusion often focus on the behavior of governments as a whole (and not individuals). We provide a template for how to study the behavioral decisions of policymakers related to policy diffusion. Our particular field experiments test whether there is a bias toward seeking out information on the policy experiences of local governments in one’s own country as opposed to governments outside of the country. The experimental evidence shows that local officials do not exhibit this bias. Further, they do not exhibit a bias against information about policies that have received an EU endorsement. Our results are a positive sign that nationalistic forces are not diminishing inter-country, policy diffusion within Europe, and our design provides a template for future experiments on policy diffusion.
Keywords
Introduction
The policy diffusion literature assumes that policymakers look to other governments to guide their own policy choices (Graham et al., 2013). There are several mechanisms that can facilitate policy diffusion across governments, including imitation, competition and learning (Gilardi et al., 2009; Meseguer, 2006; Shipan and Volden, 2008; Füglister, 2012). A key assumption of this literature, regardless of the mechanism, is that individual policymakers seek out information about the policies used elsewhere. If they do not seek out the information about the policy, it is difficult to implement the policy.
Although the key assumption in the diffusion literature deals with the behavior of individual policymakers, most tests look at the decision of the government unit and not the individual (for exceptions see Butler et al., 2017; Karch 2007). In this paper we present a field experimental design to test questions regarding the behavior of individuals in the diffusion process. To illustrate how this general approach works, we present the results of field experiments exploring if policymakers exhibit a preference for policy information from localities located in their own country. In other words, are politicians more willing to learn from the experiences of a local government in their country than they are to learn from the experiences of local governments located in other countries? Our focus has been motivated by the increasingly nationalistic tone in many developed countries. Does this nationalistic tone indicate an unwillingness on the part of policymakers to seek out information from governmental units from outside of their country?
Our evidence comes from three field experiments conducted with local municipal officials from nine European countries—Austria, Belgium, Estonia, Finland, Germany, Hungary, Italy, Netherlands and Sweden—in 2016 and 2017. In our experiments, we sent email messages to local municipal officials providing them with a link to a report about successful local waste policies that help combat global warming. For the dependent variable, we study behavioral measures of information seeking by tracking who opened the email and who clicked on the link to the original report.
In study 1, we test whether officials are biased against policies implemented in other countries by randomizing whether we stated that the report covered policies from their own country or another country. 1 By varying where the policy came from, we are able to directly measure if there is a bias against seeking information from other countries versus seeking information from local policymakers in their own country. We find no evidence that the policymakers exhibit a bias in favor of governments in their country. About 20 percent of the officials in both conditions click on the report.
In study 2, we test whether officials react negatively to policies endorsed by the European Union (EU). We focus on the effect of the EU endorsement because the EU promotes policy sharing across countries. For the study we randomized whether an EU endorsement of the report was included in the email. We find that including the EU endorsement did not affect policymakers’ interest in the policy. Finally, we conduct study 3 to mitigate concerns about self-selection bias. Study 3 looks at how delivering the treatment in the subject line of the email affected policymakers’ reactions.
Our study is important for two reasons. First, many of the foundational assumptions about policy diffusion refer to what individual policymakers do. Experiments would ideally study mechanisms at the level that they are assumed to occur. Our study provides a model for designing experiments to study policy diffusion.
Second, our results have implications for understanding cross-border interactions today. International organizations, like the EU, are designed to facilitate policy sharing and learning across borders. These organizations put substantial effort into facilitating policy diffusion. A lack of any national bias or any anti-EU bias is good news for the efforts to increase cross-border policy diffusion of international organizations. Local policymakers are not exhibiting strong nationalistic biases in their willingness to seek out information about new policies.
Are policymakers biased against policy information from other countries?
In study 1 we test whether individual, municipal, European policymakers are less likely to seek out information about policies enacted in other EU countries than they are to seek out information on policies from their own country. We carried out the experiment by emailing municipal officials with a link to a report on solid waste management strategies used by local governments, and varying whether the country mentioned in the email was their own country or another country. We then tracked which officials sought more information about the report by clicking on the link to view the report.
One of the researchers on our team emailed the message below to municipal officials in November 2016 in their local language. 2 The text in bold was randomly varied across policymakers. 3
We focus on municipal solid waste for three reasons. First, this is a relevant issue for local officials because they have power to address it. Second, this issue affects a key societal challenge, climate change, that is becoming increasingly important for mayors to deal with. 4 Third, climate change is a valence issue that cuts across party lines in Europe.
The report was created for the European Commission and made available on their website. It included information about municipal policies in the following countries: Austria, Belgium, Estonia, Finland, Germany, Hungary, Italy, Netherlands and Sweden. In the experiment, we sent an email to local officials from cities with 10,000 or more people in these nine countries.
We randomized the treatment so that half of the policymakers received a message that the report covered policies from their country and the other half received a message that the report covered policies used in one of the other eight countries. If the official was assigned to one of the other eight countries, we randomized which of the eight countries was used. 5 We block-randomized within country. We restricted the sample to local government officials in the countries mentioned in the report to avoid using deception.
Research assistants who were fluent in the language collected the contact information for policymakers in the study. They started with a list of all cities with a population of 10,000 or more, and performed a search to find the city’s website. They would then record the name and email address of the mayor (or the equivalent) from the website. We only collected the contact information of one official per city. Table 1 shows the number of municipalities and the number of individuals from each country that opened up the email.
Countries used in field experiment.
We tracked this information by using a service that tracks who opens emails by embedding a small image in the email. This information allows us to carry out a placebo design (Gerber and Green, 2012: 161–164; Nickerson, 2005, 2008). Only the people who opened the email could be exposed to the treatment (just like only those who are home during a “get out the vote” (GOTV) campaign can be exposed to the message). Limiting the sample to those who opened the email allows us to directly estimate the treatment effect on this population.
One feature of our study is that by limiting ourselves to those who opened the email, we are estimating a local average treatment effect. We are not learning the effect of the information on all officials; instead, we are learning about the effect of this information on policymakers who open their email and who do not have programs that block tracking emails. These policymakers may be systematically different. Unfortunately, no existing studies that we are aware of directly address the question of who uses email blockers. However, there are some studies that focus on human uptake of general cybersecurity measures. One of the main findings from these studies is that many people do not adopt these measures because they find them hard and confusing (Cranor, 2012; Leon et al., 2012). Those who do use cybersecurity measures are those who think the risks from not using these measures is high (Fagan and Khan, 2016).
In June 2016, the Pew Research Center conducted a Cybersecurity Knowledge survey which provides a few more insights into who is likely using email tracking blockers. In that survey, the Pew Center asked participants about factors that predicted their knowledge about cybersecurity. Among other things the survey asked individuals whether email is encrypted by default (see Olmstead and Smith, 2017). While they asked about many cybersecurity measures, this is the one most closely related to email tracking blockers. Although they only asked about knowledge and not behavior, this question is likely to be correlated with the decision to use blockers because concern about risks is a big predictor of using the associated technologies (Fagan and Khan, 2016). Thus, the people who do not think this is likely to be a concern are also the type of people who are likely to be in our sample. In this regard, the Pew Study found that age was not a factor, but that education was. More educated individuals were more likely to know that email was not encrypted by default. While 66 percent of people who finished college correctly know that email is not encrypted by default, only 44 percent of those who had only some college knew that email was not encrypted by default.
Overall, these previous studies suggest that the use of email blockers means that the sample we use in our analysis is skewed in at least two ways. First, they are likely to be less educated than the overall population of public officials in these countries. Second, they are also less concerned by cybersecurity. If these characteristics are correlated with how officials respond to the treatments in the study, then this will affect what inferences can be drawn about the larger population. In this regard, we suspect that these two characteristics work in opposite directions. We suspect that those who have less education might be less open to policies originating in other countries. In other words, the fact that our sample is likely less educated than the population might lead us to overestimate any bias. However, we suspect that the second factor, having less concern about risk, might work in the opposite direction. If you are less concerned about risk, you might be less cautious and more generally open. Having a sample that is less cautious might lead us to underestimate the bias that would be observed in the full population. We suspect that the difference in any reaction to the treatment between our sample and the full population is likely to be small enough that the results are still informative of nationalistic bias in policy learning more generally.
Before getting to the results, it is also important to note that we are using emails for our sample. Each of these emails is connected to a policymaker at the municipal level in Europe. We cannot determine who opened those emails. Most of these cities are small enough that we suspect that the email was opened by the policymakers themselves. However, even if it was opened by staff, we think these results are important. Salisbury and Shepsle (1981a, 1981b) argue that the politician’s enterprise, which includes staff members serving as agents with the politician as principal, is the relevant unit for policymaking. The very municipalities that employ staff for dealing with correspondence are also the places that employ staff more in the larger policymaking process. In that sense we are testing for bias among the policymaking enterprise. And even if the staff are only serving as screeners, this is important to learn whether bias is occurring at that stage. If the screeners are blocking some information from getting passed on because of where the policy is coming from, bias is still preventing it from having an effect on the policymaking process.
Among those who opened the email, we test whether the treatment affected their likelihood of clicking on the link to the report. We focus on email communication because it is a very common form of sharing information in the digital age. Further, we focus on clicks because it is an important form of information seeking (Ryan, 2012); this allows us to measure their learning behavior in a naturalistic setting. Those who clicked on the link were redirected to the European Commission website where the report is posted. 6
Table 2 gives the average click rate by treatment among those who opened the email. In both conditions, just over 20 percent of the policymakers clicked on the link. The difference between conditions is small (about 1 percentage point) and not statistically significant. 7 Local policymakers do not exhibit a bias against other countries (relative to their own) in their policy information-seeking behavior.
Average click rate by country treatment.
Does an EU endorsement have an effect?
International organizations like the EU can also play a role in promoting inter-country learning. Recent nationalistic movements in Europe have expressed a desire to reduce the influence of the EU in shaping their policy. Thus, rejecting policies that receive an EU endorsement is another way in which officials might exhibit a national bias in information seeking. In study 2, we test whether an EU endorsement causes officials to be less likely to seek out information about a policy (by clicking on the link to access the report). 8
The design of second study closely follows the design we used in study 1. We emailed officials with information on a policy report relevant for local government, tracking who opened the emails and who clicked on the link in order to seek out more information. The email was sent in March 2017 to the same set of policymakers collected for study 1. The email text is displayed below.
The email noted that the study came from Ireland (which is a country named in the report). We chose Ireland because it was not one of the countries in the sample and not contiguous to any of the countries in the sample. We randomized whether the email included an EU endorsement of the policy (“[t]he study was commissioned by the European Commission and is available at the European Commissions’ website”). Half of the local officials received the email with this information included and half did not.
Table 3 shows that the EU endorsement did not have a negative effect on interest. The difference is statistically insignificant and the point estimate is pointing in the wrong direction. 9 Not only do local policymakers not exhibit a bias against information on the policy used in other countries, they also do not exhibit a bias against information endorsed by the EU. If anything, these officials exhibit a positive response toward an EU endorsement.
Average click rate by EU treatment.
Potential concerns
One potential concern is that there might be a correlation between not opening the email and having a national and/or anti-EU bias. Officials with either a national or an anti-EU bias may simply not have opened their emails. A second concern relates to statistical power. Our analyses for studies 1 and 2 rely on the 800 policymakers that we know opened their emails. With 800 observations and a baseline rate of 20 percent clicking on the link (e.g. study 1), the minimal detectable effect (MDE) for a two-sided test would be 8.5 percentage points. Even if we look at the MDE for a one-sided test, which is not unreasonable given that we are testing a one-sided hypothesis, the MDE is still 4 percentage points. This means that a test with our sample size is not designed to identify an effect less than the MDE. Yet a bias of 6 or 7 percentage points would still be substantively significant. In other words, the test is underpowered for reasonably sized effects.
In order to partly address these concerns, we ran a third study. This time the treatment was included in the subject line of the email, with half being told that the report was about their own country and half being told it was about another country (the body of the email was similar to that of the EU bias study but with a link to a new report). This matters because we are now using the full sample with close to 3150 observations. 10 The MDE for a two-sided test where the baseline rate is 30 percent (see study 3 below) is 4.7 percentage points, and the MDE for a one-sided test would be 2.3 percentage points. Study 3 provides more statistical power for evaluating the bias hypothesis.
No effect on opening the emails
In study 3, we varied the subject line as follows (the text in bold was randomly varied across policymakers):
Table 4 shows that the country mentioned in the subject line had no significant impact on the likelihood of opening the email. Self-selection due to a national bias is not a major issue with the sample used for this study. We also looked at the effect of the country in the subject line on the likelihood of clicking on the report among those who opened the email. Table 5 shows that there was again no effect on the likelihood that they clicked on the report.
Average open rate by country subject-line treatment.
Average click rate by country treatment for study 3.
We also performed two other analyses that confirm the null results. First, Appendix A presents the results on a country-by-country basis (see Figures A1–A3). Those results confirm what we find here: There is no significant treatment effect overall or in any of the countries we study. Second, Appendix B presents the results of using the full sample. In other words, it presents the intent-to-treat effect. Those results indicate the effect of the overall click rates, without adjusting for those who open the emails. In the main body of the paper we have focused on the placebo design (i.e. limiting the sample to those who opened the email) because this maximizes the statistical power for finding an effect (Gerber and Green, 2012: 161–164; Nickerson, 2005, 2008). Either way, the results show there is no observed effect.
Scope conditions
Our results suggest that there is not a bias against learning from other countries. This good news must be tempered by the fact that our test is narrow in scope. We look specifically at the behavior of EU policymakers in response to information from other EU policymakers. The lack of bias against fellow European countries does not mean that there might not be bias against policy information from other countries. Figure 1 helps illustrate this point with data from a survey Pew conducted in 2017. 11
Pew 2017 survey results by country of respondent.
In that 2017 survey, Pew asked respondents in multiple countries, including five of the countries we use in this study (Germany, Hungary, Italy, the Netherlands and Sweden), about how they felt toward Germany, Britain, The EU, The US, China and Russia. Figure 1 displays the 95% confidence intervals for their responses. As the data shows, the gap between the two European countries, Germany and Britain, is much smaller than the attitudinal gap between those countries and Russia. While respondents had a somewhat favorable opinion of Germany, they had a more unfavorable opinion of Russia (and China and the US were doing only slightly better). Figure A4 in Appendix C gives the results broken down by the respondent’s country and shows that this pattern generally holds across countries. Namely, Germany, Britain and the EU are rated more favorably than the US, China and Russia. While we find that European policymakers do not exhibit a bias for information from other European countries, they might be biased if the information came from a country further afield, such as Russia.
That said, we think it is still significant that we do not find a bias within Europe. With Brexit and the electoral success of many populist parties across Europe, it was an open question of whether we would find bias. This test was directly motivated by the political events we have observed. The fact that we do not find bias, is a good sign. Future research can see whether this lack of bias also applies to countries that are less closely related.
Any future experiments can strengthen their design by strengthening the treatment. In our case, all of the emails were sent by a scholar based in Britain. We made this choice because it allowed us to hold constant the messenger and vary only the message. However, one might consider having the emails coming from scholars based inside versus outside the country. This could strengthen the treatment (although it would mean having a bundled treatment where the message and the messenger and simultaneously varied). Or, alternatively, one could hold constant the messenger, but make sure it always comes from someone within the country. Because the emails all came from a scholar based in Britain, the policymakers may have simply been responding to that signal and ignoring anything else about the content of the email.
Discussion
In a period where voters and politicians appear to be attracted to nationalist sentiment, are European policymakers open to learning from other European countries? We used field experiments to test this question and found that politicians are open. We find no evidence that policymakers exhibit a bias toward learning only from governments in their own country or against learning from the EU. This is a positive sign for local representation in Europe, where officials are acting as “enlightened statesmen,” seeking out information without bias (Madison, 1787). Moreover, this lack of bias is good news for international organizations that aim to increase policy learning across borders.
Our results also have important implications for studying policy diffusion. Prior studies have consistently shown that geographic proximity is a strong predictor of diffusion (Graham et al., 2013). Policymakers appear to be learning and implementing the policies of their neighbors. We have shown that, at least in the intra-European context, this pattern is not driven by a bias against learning from governments in other countries or international organizations. We think that exploring the causes of exposure to policy information represents one useful avenue for future research. It may be that people are differentially exposed to information from geographically more proximate governments. The advantage of an experiment is that we held constant the level of exposure. That is, we decided whether they received information about their own country or another country. In practice, however, they may naturally hear more about government policies within their own country. Often governmental groups, such as municipal leagues, are based upon geographical borders. It would not be surprising that a municipal league in Italy to focus on policies being tried within Italy when they communicate with policymakers within their organization. This would lead to an apparent preference for policies being tried within their country even if they do not exhibit bias when given equal exposure to all policies. In other words, we need to know more about what affects the information that policymakers receive and how intermediate organizations (such as municipal leagues) play a role in that process.
Supplemental Material
Appendices_for – Supplemental material for Studying policy diffusion at the individual level: Experiments on nationalistic biases in information seeking
Supplemental material, Appendices_for for Studying policy diffusion at the individual level: Experiments on nationalistic biases in information seeking by Daniel M. Butler, Catherine E. De Vries and Hector Solaz in Research & Politics
Supplemental Material
FigA1 – Supplemental material for Studying policy diffusion at the individual level: Experiments on nationalistic biases in information seeking
Supplemental material, FigA1 for Studying policy diffusion at the individual level: Experiments on nationalistic biases in information seeking by Daniel M. Butler, Catherine E. De Vries and Hector Solaz in Research & Politics
Supplemental Material
FigA2 – Supplemental material for Studying policy diffusion at the individual level: Experiments on nationalistic biases in information seeking
Supplemental material, FigA2 for Studying policy diffusion at the individual level: Experiments on nationalistic biases in information seeking by Daniel M. Butler, Catherine E. De Vries and Hector Solaz in Research & Politics
Supplemental Material
FigA3 – Supplemental material for Studying policy diffusion at the individual level: Experiments on nationalistic biases in information seeking
Supplemental material, FigA3 for Studying policy diffusion at the individual level: Experiments on nationalistic biases in information seeking by Daniel M. Butler, Catherine E. De Vries and Hector Solaz in Research & Politics
Supplemental Material
FigA4 – Supplemental material for Studying policy diffusion at the individual level: Experiments on nationalistic biases in information seeking
Supplemental material, FigA4 for Studying policy diffusion at the individual level: Experiments on nationalistic biases in information seeking by Daniel M. Butler, Catherine E. De Vries and Hector Solaz in Research & Politics
Supplemental Material
Replication_files_for – Supplemental material for Studying policy diffusion at the individual level: Experiments on nationalistic biases in information seeking
Supplemental material, Replication_files_for for Studying policy diffusion at the individual level: Experiments on nationalistic biases in information seeking by Daniel M. Butler, Catherine E. De Vries and Hector Solaz in Research & Politics
Footnotes
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.
Supplemental material
Notes
Carnegie Corporation of New York Grant
This publication was made possible (in part) by a grant from the Carnegie Corporation of New York. The statements made and views expressed are solely the responsibility of the author.
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
