Bottom-up accountability and public service provision: Evidence from a field experiment in Brazil

Abstract

Does local oversight improve public service delivery? We study the effect of a mobile phone application that allows citizens to monitor school construction projects in Brazilian municipalities. The app prompts users to submit data about construction sites, sends such crowdsourced information to independent engineers, and contacts the mayors’ offices about project delays. Our results show that the app has a null impact on school construction indicators. Additionally, we find that politicians are unresponsive to individual requests. The results question the impact of bottom-up monitoring on public service performance and suggest that interventions targeted at other groups, or focused on different issues, may produce better policy outcomes.

Keywords

Accountability Brazil impact evaluation state capacity technology

Introduction

A robust accountability system is crucial for efficient public services provision (Besley and Ghatak, 2003; Cameron, 2004; Ferejohn, 1986; O’Donnell, 1998). In its standard definition, accountability is understood as the process of holding authorities responsible for their actions (Finer, 1941; Mulgan, 2000; O’Loughlin, 1990). Past studies show that accountability has a positive impact on governance; it ensures that politicians act on behalf of voters (Freire, 2010; Moncrieffe, 1998), reduces the opportunities for rent-seeking and corruption (Deininger and Mpuga, 2005; Wenar, 2006), and improves the quality of public services (Adsera et al., 2003; Björkman and Svensson, 2009). Recent research also suggests that accountability leads to higher economic growth because it limits state discretion in the economy and increases long-term investments in human capital (Benhabib and Przeworski, 2010; Ponzetto and Troiano, 2018; Suebvises, 2018).

However, accountability mechanisms extend beyond elections. One promising model is that of bottom-up monitoring, in which citizens receive information about the shortcomings of a given project so they can evaluate and pressure underperforming public officials (Kosack and Fung, 2014; Molina et al., 2016; Raffler et al., 2018). Proponents argue that bottom-up accountability is effective because: (a) constituents have first-hand information about the outcomes of local policies; (b) citizens have incentives to attack corruption that directly affects them; and (c) policy-makers are sensitive to social punishment from their own communities (Serra, 2011: 570). In this regard, bottom-up accountability offers a potential solution to the principal-agent dilemma in public service by aligning the interests of state officials with those of the constituency they serve (Barro, 1973; Raffler et al., 2018).

Here we assess the impact of Tá de Pé (TDP), a mobile phone application designed to lower the costs of evaluating public works in Brazil. Developed by Transparência Brasil,¹ TDP allows citizens to learn the location of public school construction sites, check their completion status, and anonymously request information from competent authorities. TDP users can also take pictures of the construction sites and submit them to independent engineers for examination (see Figure 1). If the engineers classify the construction as delayed, TDP prompts users to send a message to the mayor’s office asking for completion estimates and explanations about the construction status. TDP has been online since April 2017 and was the winner of the 2016 Google Social Impact grant with more than 200,000 popular votes.²

Figure 1.

The Tá de Pé mobile phone application (a) list of school construction sites close to the users’ location; (b) the school construction selected by the user is delayed by 9 months; (c) citizens can add information to the photos they submit via the app.

We use the TDP app to conduct two experimental interventions and test the impact of citizen oversight on five outcomes related to school completion rates and complaints to public authorities. Overall, providing information to citizens has no consistent impact on policy outcomes. In the first experiment, we find that the TDP app increased the likelihood of construction cancellation by 2%, but the result does not replicate. The remaining five models have null results. In our second intervention, none of the estimations reached conventional levels of statistical significance. Importantly, all coefficients are small, which suggests that even if the TDP app had a significant effect on the outcomes, its substantive impact would be negligible.

The findings raise questions about the ability of citizens to hold representatives accountable using bottom-up monitoring. On the one hand, Björkman and Svensson (2009), Lagunes (2018), and Reinikka and Svensson (2005) report better policy outcomes after providing information to local communities. On the other hand, Banerjee et al. (2010), Keefer and Khemani (2014), Lieberman et al. (2014), Björkman Nyqvist et al. (2017), Olken (2007), and Raffler et al. (2018) find little evidence that information-based interventions lead to greater government responsiveness. Our results are in line with the latter group and suggest that local oversight is ineffective in altering government behavior in Brazil.

The Tá de Pé project

The Tá de Pé³ (TDP) cell phone application is an initiative carried out by Transparência Brasil to foster bottom-up accountability in the Brazilian public sector. More specifically, the main goal of the TDP project is to improve responsiveness in government education expenditures. The TDP app incentivizes citizens to provide up-to-date information about unfinished school constructions in their neighborhoods, and that information will be assessed by a group of independent specialists. If the construction is behind schedule, TDP provides a writing platform whereby citizens can report to public officials quickly and anonymously. The app then writes a notification to the mayor’s office, which has 15 days to reply. If the mayor’s office does not respond to the request, the app forwards the notification to the Brazilian Ministry of Education, making it harder for the municipality to access federal funds in the future. The motivation behind this intervention is that providing information to citizens empowers individuals to closely monitor public works. This, in turn, results in better social outcomes as public agents become more responsive to community demands.

Transparência Brasil built the app from January to March 2017 and tested it in May of that year. The first stable Android version was deployed on Google Play on 14 August 2017. A version for iOS came about six months later. In October 2017, Transparência Brasil started a Facebook campaign in order to publicize the app. Facebook is the most widely used social media network in Brazil with around 72 million users (Statista, 2018). The campaigns attracted 2028 new users to the platform in October 2017 only.

Transparência Brasil partnered with the Brazilian branch of Engineers without Borders (EWB), an independent non-governmental organization,⁴ to provide technical assessment of school completion rates based on user-submitted photos and GPS coordinates. The engineers’ reports are later uploaded to the TDP database and stored on the users’ mobile devices so citizens can follow the progress of the reported constructions.

The TDP also received feedback from Brazilian computer scientists and policy analysts. In 2017 and 2018, Transparência Brasil announced two team programming competitions, called “Tá de Pé Hackathons”, where contributors could fix code bugs and suggest new functionalities to the TDP project. One of these innovations consists of a Twitter bot (https://twitter.com/tadepeapp) which posts a message on the social network each time a user submits a new picture for evaluation or a municipality responds to a citizen’s request. This allows any interested parties, including those who do not use the TDP app, to check the state of school construction sites.

Experimental design

Between August 2017 and July 2019, we implemented two interventions to measure the effect of the TDP app on five school construction outcomes plus a placebo test. The outcomes are: (a) a placebo outcome indicating the percentage of the project completed before the impact evaluation started; (b) the percentage of the project completed by the end of the interventions; (c) the difference between the percentage reported as completed before and after the interventions; (d) the number of finished constructions; (e) the number of cancelled constructions; and (f) the number of schools where construction companies updated the conclusion dates. Table 1 depicts the expected effects for each of the studied outcomes.⁵

Table 1.

Outcomes and expected effect of the TDP intervention.

	Expected impact	Meaning
Placebo: Percentage of the project completed before the impact evaluation started	Null	The placebo outcome, as reported before the intervention, should have a null impact. This represents the absence of differences between treatment and control prior to the intervention.
Outcome 1: Percentage of the project reported as completed by the end of the intervention period	Positive	If the intervention has a positive effect, the firms should increase their efforts toward finishing the construction more quickly
Outcome 2: Difference between the percentage reported as completed before and after the intervention	Positive	If the intervention has a positive impact, the difference between before and after the intervention should reflect this
Outcome 3: Dummy indicator for finished constructions	Positive	If the intervention has a positive effect, more schools should be reported as finished in the treatment group
Outcome 4: Dummy indicator for cancelled construction	Negative	If the intervention has a positive impact, fewer constructions should be abandoned and reported as finished in the treatment group
Outcome 5: Number of schools where construction companies updated the conclusion dates	Positive	If the intervention has a positive effect, firms and mayor’s offices should be responsive to the public, and update their finishing dates

The first intervention was carried out from August 2017 to July 2018 using the Android version of TDP. The randomization was conducted at the municipal level. We randomly selected 344 municipalities to the control group and included 2642 in the treatment group. Our control condition consists in removing all information about school construction from the TDP app in the chosen municipalities, so that citizens were unable to report constructions in the control municipalities.

To evaluate the random assignment, we used the following pre-treatment variables: (a) log of municipal population in 2015; (b) log of number of poor families in each city; (c) log of total federal transfers to the municipality in 2016; (d) federal government indicator for primary school quality; and (e) federal government indicator for secondary school quality. The data come from the Brazilian Ministry of Education and the 2010 Brazilian Census. Balance tests show that the randomization was successful and further details are available in the supplementary material online.

We also conducted two manipulation checks and analyzed the number of TDP app downloads by municipality and over time. Figure 2 displays the results and indicates that the treatment has good territorial variability. There are 455 downloads in the 1023 municipalities in the treatment condition. Downloads peak during the Facebook TDP campaign, launched in October 2017, then diminish in the following months. Overall, 6092 users downloaded the app during intervention 1.

Figure 2.

Manipulation checks for intervention 1 (a) geographical distribution of the treatment condition; (b) number of TDP app downloads from August 2017 to July 2018.

The second intervention is similar to intervention 1 in all but three characteristics. First, the TDP app was then available for both Android and iOS devices. Second, we randomized the intervention at the school level, with 659 control and 3717 treatment units. We used blocked randomization stratified by Brazilian states, school construction status (under construction, stopped, unfinished), and whether the municipality spent more on school construction than the distribution median. Finally, the intervention period lasted from August 2018 to July 2019.

Balance tests and manipulation checks were also successful for intervention 2. In total, 443 municipalities downloaded the app. There were about 1000 user downloads in August 2018, right after intervention 2 started, and a second spike around December (see Figure 3). The app gained 4078 new users during intervention 2. The number of downloads is smaller in this second intervention as there was no associated social media campaign in that period.

Figure 3.

Manipulation checks for intervention 2 (a) geographical and (b) temporal variation of TDP app downloads from August 2018 to July 2019.

Data from Google Analytics suggest that users did engage with the TDP app. On average, each user launched 60 app sessions, which indicates their interest in the application. In total, the app had 53,928 screen visualizations, with an average of 2.42 screen visualizations per session.

We estimate all models using the following regression equation:

Y_{i} = α + β T_{i} + γ X_{i} + θ Z_{i} + ε_{i}

where i indexes the experiment units. Y_i is one of the six outcomes described above, α is the intercept, β denotes the average treatment effect, and T_i is a binary treatment indicator. γ is a vector of fixed effects, X_i is a matrix of Brazilian states’ fixed effects, θ is a vector of controls, and Z_i an array of controls for the case i. The error term is denoted by ε_i. We cluster the standard errors at the municipality level as mayors are responsible for school investment decisions in Brazil.

Results

Table 2 summarizes the main results of intervention 1. Each column represents the treatment effect of the TDP app on one of the outcomes we measured for this study. All models reported here include the five control variables described in the previous section and Brazilian states’ fixed effects. We also estimated the models without control variables, without fixed effects, and with nearest-neighbor matching. The results are very similar to those below.

Table 2.

Impact evaluation: intervention 1.

	Dependent variable
	Investment before	Investment after	Delta investment	Finished construction	Cancelled construction	Updated date
	(1)	(2)	(3)	(4)	(5)	(6)
ATE	−0.99	−1.12	−0.13	0.002	0.02	0.05
	(2.59)	(2.96)	(1.03)	(0.01)	(0.01)	(0.05)
Controls	Yes	Yes	Yes	Yes	Yes	Yes
State fixed effects	Yes	Yes	Yes	Yes	Yes	Yes
Observations	2986	2986	2986	2986	2986	2926
R	0.16	0.14	0.05	0.02	0.07	0.13

Note: p<0.1; p<0.05; p<0.01.

Cluster-robust SEs at the municipality level. ATE: average treatment effect.

We find that the app only has a small effect on cancellation rates. The TDP application increases the likelihood of cancelling the construction by 2.07%. While this result goes in the opposite direction of our theoretical expectations, the finding is inconsistent and does not replicate in the second experiment. All other coefficients are not statistically significant at conventional levels. On the one hand, the results indicate that our placebo outcome, the percentage of the projects completed before the intervention, indeed behaves as predicted. On the other hand, we expected the five remaining outcomes to improve after the introduction of the app. The literature on bottom-up accountability argues that delivering more information about the shortcomings of public services provision will put citizens in a position where they can monitor state agents and improve provider behavior (Raffler et al., 2018). Our results do not lend support to that hypothesis.

Table 3 shows the results of the second intervention. The treatment does not have a statistically significant effect on any of our outcomes of interest, including the placebo. This raises further questions about the effect of the TDP app on school completion. Note that the effect signs are also inconsistent with improving the outcomes, which demonstrates that our results do not derive from low statistical power or the reduced control group size.

Table 3.

Impact evaluation: intervention 2.

	Dependent variable
	Investment before	Investment after	Delta investment	Finished construction	Cancelled construction	Updated date
	(1)	(2)	(3)	(4)	(5)	(6)
ATE	−1.33	−2.26	−0.94	−0.001	0.01	0.002
	(1.45)	(1.56)	(0.61)	(0.01)	(0.01)	(0.02)
Controls	Yes	Yes	Yes	Yes	Yes	Yes
State fixed effects	Yes	Yes	Yes	Yes	Yes	Yes
Observations	3226	3226	3226	3226	3226	3109
R	0.12	0.11	0.04	0.02	0.17	0.09

Note: p<0.1; p<0.05; p<0.01.

Cluster-robust SEs at the municipality level.

Inverse probability weights computed by the randomizr R package. ATE: average treatment effect.

Figure 4 shows the results of our randomization inference tests. Randomization inference allows us to estimate the probability of the sharp null hypothesis over all possible randomizations that could have occurred under our research design (Coppock, 2019; Gerber and Green, 2012). We fail to reject the null in all but the finished school indicator in experiment 1.

Figure 4.

Sampling distribution of the estimated coefficient for our six outcomes in two interventions. Graphs on the left correspond to randomization inference estimates for intervention 1 and those on the right describe the results for intervention 2.

We also note that effect sizes are small in all estimations and that the coefficients flip signs in all but one of our six dependent variables. This provides further evidence for the null results: not only would the app have a low impact even if the treatment were significant, but the results could go against what the bottom-up accountability theory predicts. At least in the school construction outcomes we investigate here, we find little evidence that grassroots monitoring works in the context of school constructions in Brazil.

Discussion

In this paper, we discuss whether delivering information to citizens via a mobile phone application fosters community oversight and political accountability in Brazil. Our two interventions show that the results are at best mixed. Although we find some treatment effect on school cancelling rates in the first intervention, the app has no consistent impact on our outcomes of interest. These findings add to the studies that cast doubts on the relationship between bottom-up accountability and local policy performance (e.g. Banerjee et al., 2010; Lieberman et al., 2014; Raffler et al., 2018).

What factors, then, are driving these results? It seems unlikely that the null results derive from flaws in the research design. First, our study is well powered. Although the treatment is indirect—the person has to download the app, find a school construction, and then report it—we included a substantial number of schools in the treatment groups. Second, balance and manipulation tests indicate that the treatment allocation was successful, so we can rule out problems in the randomization procedures. Third, data from Google Analytics confirm that citizens indeed used the app and provided information to our dataset. This indicates that the treatment manipulation was effective. In this sense, it is unlikely that our results derive from low user response. Fourth, after doing a series of robustness tests, we still find no firm evidence of treatment effect. Finally, note that the signs of the coefficients are frequently contrary to our theoretical expectations. This rules out a possible concern about statistical power with our small control group approach.

We discuss some possible reasons why community monitoring did not work in our case. One plausible explanation is that individuals were unable to differentiate the effect of political corruption from those of spending cuts. Due to the severe economic crisis in 2014–2016, the Brazilian federal government introduced discretionary spending limits that affected public investment (Rossi and Dweck, 2016). Politicians may argue that delays in school constructions are not derived from their misuse of government funds but from the austerity measures. If this is the case, citizens will not blame local politicians for the underprovision of public goods. Consequently, representatives can dismiss individual requests as the issue is unlikely to escalate.

The electoral cycle might also have decreased the potential effect of the treatment. As the experiment was fielded right after Brazil’s municipal elections, incumbents might have disregarded the requests because they did not see the demands as politically costly in the short run. Having just taken office, mayors might have focused their attention on the formation of government coalitions or on budget concerns. Future research may evaluate how electoral dynamics interact with citizen oversight, potentially by replicating informational experiments in different stages of the political cycle.

In sum, our experiments suggest that popular participation and bottom-up monitoring may not be effective to improve public service delivery in the case of school constructions in Brazil. Nevertheless, the null findings are informative to researchers and policy-makers. The most important recommendation derived from this study is that interventions targeting elite groups, such as lobbyists or civil servants, might render better school construction outcomes than those focused at the community level. Another core lesson is that although digital interventions are promising means to deliver information, perhaps they do not have the same impact as personal, face-to-face communication. Since many developing countries share Brazil’s issues with education provision, the shortcomings we describe here serve as warnings for future interventions. Finally, whether this study generalizes beyond school constructions to other bottom-up programs, and to other contexts, remains to be studied.

Supplemental Material

appendix – Supplemental material for Bottom-up accountability and public service provision: Evidence from a field experiment in Brazil

Supplemental material, appendix for Bottom-up accountability and public service provision: Evidence from a field experiment in Brazil by Danilo Freire, Manoel Galdino and Umberto Mignozzetti in Research & Politics

Footnotes

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iDs

Danilo Freire

Umberto Mignozzetti

Supplementary materials

The supplementary files are available at .

The replication files are available at .

Carnegie Corporation of New York Grant

This publication was made possible (in part) by a grant from the Carnegie Corporation of New York. The statements made and views expressed are solely the responsibility of the author.

Notes

References

Adsera

Boix

Payne

(2003) Are you being served? Political accountability and quality of government. The Journal of Law, Economics, and Organization,19(2): 445–490.

Banerjee

Banerji

Duflo

, et al. (2010) Pitfalls of participatory programs: Evidence from a randomized evaluation in education in India. American Economic Journal: Economic Policy 2(1): 1–30.

Barro

(1973) The control of politicians: An economic model. Public Choice 14(1): 19–42.

Benhabib

Przeworski

(2010) Economic growth under political accountability. International Journal of Economic Theory 6(1): 77–95.

Besley

Ghatak

(2003) Incentives, choice, and accountability in the provision of public services. Oxford Review of Economic Policy 19(2): 235–249.

Björkman

Svensson

(2009) Power to the people: Evidence from a randomized field experiment of a community-based monitoring project in Uganda. Quarterly Journal of Economics 124(2): 735–769.

Björkman Nyqvist

de Walque

Svensson

(2017) Experimental evidence on the long-run impact of community-based monitoring. American Economic Journal: Applied Economics 9(1): 33–69.

Cameron

(2004) Public accountability: Effectiveness, equity, ethics. Australian Journal of Public Administration 63(4): 59–67.

Coppock

(2019) ri2: Randomization Inference for Randomized Experiments. R package version 0.1.2. Available at: https://CRAN.R-project.org/package=ri2 (accessed 13 March 2020).

10.

Deininger

Mpuga

(2005) Does greater accountability improve the quality of public service delivery? Evidence from Uganda. World Development 33(1): 171–191.

11.

Ferejohn

(1986) Incumbent performance and electoral control. Public Choice 50(1): 5–25.

12.

Finer

(1941) Administrative responsibility in democratic government. Public Administration Review 1(4): 335–350.

13.

Freire

(2010) Nova Gestão Pública e Democracia no Brasil. In: Erkens

(ed.) Nova Gestão Pública. São Paulo: Friedrich Naumann Stiftung, 9–16.

14.

Gerber

Green

(2012) Field Experiments: Design, Analysis, and Interpretation. New York: WW Norton.

15.

Keefer

Khemani

(2014) Mass media and public education: The effects of access to community radio in Benin. Journal of Development Economics 109: 57–72.

16.

Kosack

Fung

(2014) Does transparency improve governance? Annual Review of Political Science 17: 65–87.

17.

Lagunes

(2018) Guardians of Accountability: A Field Experiment on Corruption and Inefficiency in Local Public Works. Available at: https://www.uh.edu/hobby/_images/events/lagunes_perustudy.pdf (accessed 13 March 2020).

18.

Lieberman

Posner

Tsai

(2014) Does information lead to more active citizenship? Evidence from an education intervention in rural Kenya. World Development 60: 69–83.

19.

Molina

Carella

Pacheco

, et al. (2016) Community monitoring interventions to curb corruption and increase access and quality of service delivery in low-and middle-income countries: A systematic review. Campbell Systematic Reviews 12(1): 1–210.

20.

Moncrieffe

(1998) Reconceptualizing political accountability. International Political Science Review 19(4): 387–406.

21.

Mulgan

(2000) ‘Accountability’: An ever-expanding concept? Public Administration 78(3): 555–573.

22.

O’Donnell

(1998) Horizontal accountability in new democracies. Journal of Democracy 9(3): 112–126.

23.

Olken

(2007) Monitoring corruption: Evidence from a field experiment in Indonesia. Journal of Political Economy 115(2): 200–249.

24.

O’Loughlin

(1990) What is bureaucratic accountability and how can we measure it? Administration & Society 22(3): 275–302.

25.

Ponzetto

Troiano

(2018) Social capital, government expenditures and growth. Technical report 24533, Cambridge: National Bureau of Economic Research, pp. 1–56.

26.

Raffler

Posner

Parkerson

(2018) The Weakness of Bottom-Up Accountability: Experimental Evidence from the Ugandan Health Sector. Available at: https://bit.ly/2LPL6Nm (accessed 13 March 2020).

27.

Reinikka

Svensson

(2005) Fighting corruption to improve schooling: Evidence from a newspaper campaign in Uganda. Journal of the European Economic Association 3(2–3): 259–267.

28.

Rossi

Dweck

(2016) Impactos do Novo Regime Fiscal na Saúde e Educação. Cadernos de Saúde Pública 32: 1–5.

29.

Serra

(2011) Combining top-down and bottom-up accountability: Evidence from a bribery experiment. The Journal of Law, Economics, & Organization 28(3): 569–587.

30.

Statista (2018) Number of Facebook users in Brazil from 2017 to 2023 (in millions). Available at: https://www.statista.com/statistics/244936/number-of-facebook-users-in-brazil/ (accessed 13 March 2020).

31.

Suebvises

(2018) Social capital, citizen participation in public administration, and public sector performance in Thailand. World Development 109: 236–248.

32.

Wenar

(2006) Accountability in international development aid. Ethics & International Affairs 20(1): 1–23.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

1.86 MB