Outsourcing accountability: Extractive data practice and inequities of power in humanitarian third-party monitoring

Abstract

Since the early 2010s, humanitarian donors have increasingly contracted private firms to monitor and evaluate humanitarian activities, accompanied by a promise of improving accountability through their data and data analytics. This article contributes to scholarship on data practices in the humanitarian sector by interrogating the implications of this new set of actors on humanitarian accountability relations. Drawing on insights from 60 interviews with humanitarian donors, implementing agencies, third-party monitors and data enumerators in Somalia, this article interrogates data narratives and data practices around third-party monitoring. We find that, while humanitarian donors are highly aware of challenges to accountability within the sector, there is a less critical view of data challenges and limitations by these external firms. This fuels donor optimism about third-party monitoring data, while obscuring the ways that third-party monitoring data practices are complicating accountability relations in practice. Resultant data practices, which are aimed at separating data from the people involved, reproduce power asymmetries around the well-being and expertise of the Global North versus Global South. This challenges accountability to donors and to crisis-affected communities, by providing a partial view of reality that is, at the same time, assumed to be reflective of crisis-affected communities’ experiences. This article contributes to critical data studies by showing how monitoring data practices intended to improve accountability relations are imbued with, and reproduce, power asymmetries that silence local actors.

This article is a part of special theme on Commodifying Compassion in the Digital Age. To see a full list of all articles in this special theme, please click here: https://journals.sagepub.com/page/bds/collections/commodifying_compassion

Introduction

Over the past two decades, humanitarian intervention has been undergoing a transformation with the use and promotion of data, from within early warning systems, to implementation, to monitoring and evaluation (Burns, 2015; Johns, 2023). As an example of the sector's growing data appetite, the 20,400 datasets of the United Nations Office for the Coordination of Humanitarian Affairs (OCHA)'s Humanitarian Data Exchange were downloaded over 1.8 million times in 2022 – a growth of 8% on the year prior (OCHA et al., 2023: 4). More data and digitized data are promoted and used to facilitate multiple, even competing, aims. Data is seen to help enable feedback from affected communities to humanitarian agencies and funders (Madianou et al., 2016). Equally, data through digital technologies facilitate remote management, a form of ‘digital distancing’ to sustain activities in unstable or high-risk environments (Duffield, 2016).

This article is driven by an interest in the implications of an increased emphasis on data on humanitarian accountability, specifically within monitoring and evaluation practices intended to enable greater transparency between aid recipients and funders of humanitarian activities. In monitoring and evaluation, increased data use has been accompanied by the entrance of a new set of firms into the sector, known as ‘third-party monitoring firms’. Bilateral donors, United Nations (UN) agencies and large non-governmental organizations contract private, often for-profit, entities to carry out monitoring activities (UN in Afghanistan Risk Management Unit, 2015), especially in insecure operational environments where they lack in-person access.

While research has explored increased data use in humanitarian management, feedback and implementation (e.g., Burns, 2015; Fast, 2023; Squire and Alozie, 2023), there has been little interrogation of data practices in monitoring and evaluation, despite the disruption posed by a new set of actors. Yet, as scholars, including Madianou et al. (2016), have argued, elsewhere in the sector increased data use is complicating and accentuating power asymmetries. This is visible in the tension between what humanitarian actors claim data and digital technologies can do – for example, improve accountability to crisis-affected people – and what they actually do, such as restrict the channels used to listen to crisis-affected people.

Recognizing this tension, this paper considers: how are data practices employed by third-party monitoring firms affect accountability relations in the humanitarian sector? We interrogate how third-party monitoring firms and their data practices are imagined to strengthen accountability relations, the extent to which practices fulfil these aspirations, and why. We draw on the concepts of ‘technocolonialism’ (Madianou, 2019) and ‘humanitarian ignorance’ (e.g., Cheesman, 2024) in analyzing 60 interviews with humanitarian donors, third-party monitoring firms, humanitarian agencies and local enumerators with a focus on Somalia. We identify a contradiction within the perspectives held by humanitarian donors. On the one hand, we find a heightened awareness of challenges in the sector around realizing meaningful accountability to crisis-affected people; on the other hand, there is a lack of critical analysis of data practices. We suggest that this combination of informed and uninformed perspectives fuels donor optimism that third-party monitoring data necessarily improves accountability. It also obscures how third-party monitoring data practices are complicating accountability relations. Data practices are aimed at separating data from the people involved, which hides the roles of local actors, relationships and insecurities in data creation. This results in a partial view of data, and reproduces global inequalities and hierarchies of knowledge.

Literature review

This paper contributes to scholarship interested in politically aware and contextually situated conceptualizations of data, as well as studies on data use and digital technologies in the humanitarian sector. This section discusses how we conceptualize data and data practices, before looking at how data has been studied within the humanitarian sector in relation to accountability. It concludes by identifying third-party monitoring as a gap in scholarship on humanitarian data practices.

Conceptualizing data and data practices

We take as our starting point that data is invested with technical and normative characteristics. For example, ‘big data’ is often distinguished by technical characteristics of high volume, velocity (i.e., in ‘near to’ real time), and variety, which require new forms of data analysis (Kitchin, 2014; Madsen et al., 2016). Also, ‘big data’ is contextually contingent; it is created and circulates through an interplay of technological, scientific, and cultural factors (Rieder and Simon, 2016).

Beyond the humanitarian sector, scholarship in critical data studies has identified a set of persistent mythologies about what data represents and what it can achieve in knowledge production. More data and more granular data are seen to automatically improve insights and answers (Elish and boyd, 2018: 59; see also Gitelman and Jackson, 2013), with analyses of (increasingly) large quantities of digital data considered to be more objective and more neutral (Rieder and Simon, 2016) than ‘individual human or organizational decision-making processes’ (Elish and boyd, 2018: 73–74). Often, concerns about bias in data analysis are met in practice with efforts to collect more data based on the view that larger data sets can reduce bias and error (Leonelli, 2014). The institutions that deploy these methods are assumed to have integrity and independence (Van Dijck, 2014: 204).

The gap between dominant views about the value of data and data practices is well established, with studies explicating this gap in different sectors and disciplines (Kitchin, 2014). We take a broad view of data practices (Aitken et al., 2020; Choroszewicz, 2022), considering data production, sharing, interpretation and analysis, data management systems as well as its visualization, reporting and deletion. These diverse practices shape and are shaped by, narratives about data. Logics that surround data inform what questions are asked, processes designed to answer these questions, and what this means for what (and whose) knowledge can be expressed (Hartong and Förschler, 2019). Scholars’ awareness of data's situated nature has led to arguments for more contextually informed interrogations of the knowledge represented and used as data, and what this means for social, political and economic relations (Dalton and Thatcher, 2015; Elish and boyd, 2018).

Data narratives and practices in the humanitarian sector

Claims about the value of data have found a receptive audience in the humanitarian sector (Burns, 2015). Data has been elevated as a means to strengthen accountability of aid agencies – both to those who fund activities and to communities targeted by relief efforts.

There are two concurrent exercises in accountability at work in the humanitarian sector with contrasting levels of resources and critical attention devoted to them. The first is accountability from relief providers to their donors that provide the necessary funding and resources, the second is accountability to the crisis-affected communities whom projects are supposed to benefit (Madianou et al., 2016). There has been ongoing discomfort, and tensions, for humanitarian donors and agencies in managing both forms of accountability despite high-level commitments to improve the latter in policy processes including the ‘Grand Bargain’ of the 2016 World Humanitarian Summit. Williams (2010) argues ultimate accountability of humanitarian delivery to local communities, for example, in ways that close feedback loops and modify programming, is not necessarily aligned with accountability to donors over funding, activities and outputs, and therefore cannot be assumed to take place at all. Barbelet et al. (2021) and Hilhorst et al. (2021) have shown how improvements to accountability have historically tended towards the concerns of those who fund and commission humanitarian projects and evaluations, with perspectives of crisis-affected communities considered within the framework of donors’ concerns.

Existing research repeatedly reveals how data and digital technologies reproduce and exacerbate power imbalances in the humanitarian sector. This challenges accountability to the communities who are intended to benefit. Burns (2015) provides one of the earlier analyses of humanitarian data practices, looking at how spatial data has been treated as value-free in humanitarian narratives but has social and political implications. Squire and Alozie (2023) argue humanitarian data practices are marked by a logic of extraction and paternalistic discourse. They argue extractivism and paternalism devalue internally displaced people, failing to conceptualize them as knowing subjects. Fast (2023), focusing more directly on accountability relations, argues that data-sharing practices concentrate power amongst donors and humanitarian organizations based in the Global North.

Power asymmetries are also identified among scholars looking at the use of digital technologies (e.g., Duffield, 2016; Madianou et al., 2016, Sandvik et al., 2017). Madianou (2019) introduces the concept of ‘technocolonialism’ to unpick power asymmetries around the use of digital technologies in the humanitarian sector and interrogate their foundations in colonialism. These power dynamics are argued to inform the use of digital technologies in different areas of intervention. For example, technologically mediated feedback channels have been found to be oriented to donors as opposed to enabling the ‘voice’ of crisis-affected communities on their terms and in their preferred medium (Madianou et al., 2016). Looking at remote management and digital technologies, Duffield also highlights global power asymmetries, concluding this combination has enabled ‘the global North's atmospheric ability to digitally rediscover, remap and, importantly, govern anew a now distant South’ (Duffield, 2016: 149).

Power asymmetries in practice contrast with a ‘progressivist and deterministic’ narrative about digital technology among humanitarian donors and agencies (Madianou, 2019, see also Obrecht and Warner, 2016). Madianou (2019) suggests a series of logics underpin how technologies are being adopted in the sector, including logics that assume digital technologies enable crisis-affected communities to hold aid agencies to account; improve audit; enable the entry of business interests; solve complex problems; and support securitization to protect sovereignty.

The concept of ‘humanitarian ignorance’ has been utilized to explain the disjuncture between progressive narratives and uneven realities of technology. Ignorance, as a state of non-knowledge (Fejerskov et al., 2024), ‘involves omissions and gaps but is also actively produced, surfed, and shaped through acts of neglecting, forgetting, excluding, refusing, or denying’ (Cheesman, 2024: 6). Fejerskov et al. (2024) argue that ignorance helps to explain the impacts of datafication on the humanitarian sector, as it reveals how datafication results in new unknowns. Cheesman (2024) applies ignorance to the case of a humanitarian blockchain product. Cheesman identifies specific processes through which ignorance is produced and sustained: through views of digital technologies as conceptually illusive and framed by an assumed lack of understanding; misdirection from negative effects; and ‘dislocations from knowledge’ such as precarious job contracts or hierarchies for data validation. Interrogation of forms of humanitarian ignorance complements studies on the implications of data and/or digital technologies by unpacking how contradictory narratives and realities are sustained.

Third-party monitoring as a distinct set of humanitarian data practices

While signaling a new set of data-driven actors in humanitarian intervention, third-party monitoring firms and their data practices have not received any direct attention in scholarship. There are studies on the involvement of larger tech firms in the humanitarian sector (e.g., Henriksen and Richey, 2022; Martin, 2023), but no direct analysis of the private consultancy firms brought in for data analytics related to accountability. Further, few, if any, studies look specifically at monitoring and evaluation, despite knowledge production – and data practices – being central to this activity.

Monitoring and evaluation in the humanitarian sector has become a key arena for increased data use in the sector, specifically with the entrance of third-party monitoring firms for whom data is a selling point. The introduction of third-party monitoring firms is part of humanitarian donors’ response to a growing awareness that they face a knowledge gap about program implementation and outcomes, especially in high-risk contexts where they often operate remotely. Prior to the 1990s, there was little systematic evaluation of humanitarian activity, and the use of third-party evaluators was uncommon. While evaluation was a feature of economic development interventions, practitioners generally viewed it as a less rigorous and useful tool in humanitarian relief given the time-sensitive nature and insecurity of humanitarian crises (Polastro, 2014). This shifted following an especially disastrous refugee aid response in the camps surrounding Goma in then-Zaire following the Rwandan genocide of 1994. Judged as ‘among the most flagrant abuses of international relief in modern times’ in an influential assessment by Rakiya Omaar and Alex de Waal (see Parker, 2017), the response marked a key moment in the sector's drive towards a greater emphasis on quality and standards. As a consequence, donors paid more attention to monitoring and evaluation as a mechanism to improve the accountability of humanitarian agencies to crisis-affected people (Deloffre, 2016; Metcalfe-Hough et al., 2021). ¹ This occurred alongside the rise of the ‘accountability to affected people’ agenda in the sector, in which accountability to crisis-affected communities was built into terms of reference for humanitarian country teams, and featured in the 2016 Grand Bargain commitments agreed by a group of large donors and humanitarian actors (Metcalfe-Hough et al., 2021). This agenda has drawn attention to the value of two-way communication, whereby information from people affected by crises informs modifications to humanitarian response (Holloway and Lough, 2020; Warner, 2017).

The entrance of third-party monitors was a distinct moment within the increased emphasis on accountability. It signaled a particular logic, whereby consultancy firms that were experts in monitoring and evaluation were seen to be potentially ‘better’ at evaluation than humanitarian actors that might have an intimate knowledge and presence in humanitarian contexts but also a stake in the intervention. By the early 2010s, following the broader trend in the humanitarian sector toward the funding of for-profit consultancies (Richey and Ponte, 2014), it was more commonplace for donors to commission external private firms to monitor and evaluate programs including through large multi-year contracts. ²

Data has been central to the offer of third-party monitoring firms. They have been assumed to bring access to new technical tools and capabilities to collect, manage and analyze data (International Committee of the Red Cross, 2020), irrespective of whether their technology is genuinely innovative (Murphy et al., 2022). As of the early 2020s, Larissa Fast suggests:

‘Donors often outsource to third parties to monitor programs, for which third parties request and often collect the most detailed and sensitive data. While their work is legitimate, there is a lack of understanding about third-party monitors and how they collect, use, and store data, a topic that surfaced repeatedly across interviews’ (Fast, 2022: 24).

While third-party monitors do not need to be private actors, the activity has become increasingly professionalized and occupied by Global North-based private firms (Diepeveen et al., 2022). Humanitarian donors and UN agencies increasingly work with private firms, with estimates of fund to private firms amounting to US$6.6 billion between 2011 and 2015 (Murphy et al., 2022: 4). The UN OCHA states that it ‘strongly believes in the power of private sector engagement and has a long history of working with the private sector’ (2024a).

Third-party monitoring firms are inserted into existing relationships within the sector. Adapting a figure by Westphal and Meier (2020), we outline the actors implicated in third-party monitors’ data practices, identified through our interviews and literature analysis.

Illustrated in Figure 1, donor-contracted third-party monitoring firms often subcontract local third-party monitoring firms based within (insecure) humanitarian contexts who in turn work with data enumerators. Local authorities and host governments mediate activity and access of third-party monitoring firms, local partners, enumerators and humanitarian agencies.

Figure 1.

Key actors involved in third-party monitoring data practices.

Given existing scholarship has shown how digital technologies and data are (re)producing humanitarian ignorance and power inequalities, the introduction of third-party monitoring firms, especially given their complex interactions with others in the sector through data practices, requires interrogation. To what extent do third-party monitoring data practices result in power asymmetries? Are the narratives about third-party monitoring premised upon the reproduction of forms of humanitarian ignorance? Underneath these narratives, what are the implications of third-party monitoring data practices on accountability relations?

Approach and methodology

This paper investigates the relationship between third-party monitoring data practices and accountability relations in the humanitarian sector in the case of Somalia. It examines (i) narratives around third-party monitoring employed by different actors; (ii) data practices that sit behind these narratives, and (iii) what both imply for accountability between donors, humanitarian agencies and crisis-affected communities.

As mentioned above, we consider data practices broadly, including diverse ways that people interact with data, from data creation to sharing, interpretation and analysis, management, visualization, reporting and deletion. We consider who and which institutions are involved, and their relationships with one another and with data. We also explore how interviewees discussed and approached data and new data analytics. We start from a similar view of humanitarian accountability to Madianou et al. (2016), considering how the sector is framed by concerns for multiple accountability relationships: to donors about inputs and intended outputs, and to crisis-affected people, who are the intended beneficiaries.

Results are based on a qualitative study in Somalia. We conducted 60 key informant interviews in 2022 and 2023 with humanitarian accountability experts and practitioners, donors, implementing agencies, international and domestic third-party monitoring firms, and local enumerators. Starting from actors identified in academic and grey literature (e.g., Fast, 2023; Westphal and Meier, 2020), we interviewed and mapped actors involved in third-party monitoring data practices. We used a snowballing approach with interviewees to identify and speak with additional key actors involved.

Interviewees were asked how they self-identified their role in relation to humanitarian projects in Somalia. This included interviews with senior staff from U.S. Agency for International Development (USAID) and its Bureau for Humanitarian Assistance (BHA), the UK Foreign, Commonwealth & Development Office (FCDO) (formerly Department for International Development [DFID]), the World Food Program, UNHCR, the International Organization for Migration, the Swiss Federal Department of Foreign Affairs, Christian Aid and the International Committee of the Red Cross, considering their experiences and beliefs about knowledge generation through data from third-party monitoring. We also interviewed sub-contracted Somali-based monitoring and data collection firms and Somali-based enumerators. We focused on experiences around some of the larger donor-funded third-party monitoring projects in Somalia in the mid-2010s to early 2020s, including by USAID and DFID/FCDO, and looked in-depth at the DFID/FCDO program, ‘Monitoring and Evaluation of the Somalia Humanitarian, Resilience and Health Programs (MESH)’. Monitoring and Evaluation for Somalia Humanitarian Program is a large-scale and multi-year commercially contracted third-party monitoring and evaluation program. According to one interviewee, third-party monitoring was a condition of long-term DFID funding in Somalia in 2013–2014. Monitoring and Evaluation for Somalia Humanitarian Program focused on verifying where money was going, if assistance was received by intended recipients, and the extent to which aid recipients were informed about what to expect. It involved field monitoring, data collection from partners and through mobile phones, a call center and evaluations.

Our interviews were conducted in Somali and English, depending on interviewees’ preferences, and through a combination of face-to-face and remote interviews. We continued to interview individuals from different roles until achieving saturation in the types of insights shared. Interviews asked about wider experiences with humanitarian accountability efforts, as well as specific perspectives on third-party monitoring data practices in Somalia. All interviews with Somali enumerators and third-party monitoring firms were conducted in-person in Mogadishu.

We transcribed and coded interviews along three dimensions. First, we coded according to the normative language used to discuss third-party monitoring firms and data, looking for patterns and divergences in how they were discussed and by whom. Second, we organized interview content according to the discussion of the roles and experiences of the actors involved. This included descriptively, what different actors were perceived to do, as well as normatively, how interviewees judged the roles of different actors (e.g., external firms, Somali enumerators, and so on). Third, we mapped how interviewees discuss theirs, and others’, relationships, as realized through data practices. This included mapping who was seen to exercise control over whom in their interactions with data and how (e.g., formal and informal constraints), and potential ways such controls were evaded. Through this, we explored the relationships between narratives and practices, how they intersect and inform one another, and what this means for the extent to which data practices support accountability towards donors and aid recipients. Our approach to analysis was informed by scholarship on power relations, and its linkages to knowledge and ignorance in humanitarian data practices. These provided a structure for interrogating how third-party monitoring practices are affecting – and potentially obscuring – accountability relations between donors, humanitarian implementing organizations and crisis-affected people. Interviews were anonymized by individual and organization to facilitate trust and allow us to gain insights into recent and ongoing monitoring projects.

We recognize two key limitations of the insights generated through our approach. First, during our interviews, many respondents described how aid recipients had interview fatigue due to repeated interviewing as part of humanitarian activities, such as for needs assessments, monitoring and evaluation. For this reason, we decided not to interview aid recipients. Second, due to confidentiality concerns, we were not able to access and review most third-party monitoring reports to donors. Still, by interviewing participants in data practices, our study provides a unique perspective into data processes as opposed to products.

Results

The rise of third-party monitoring for humanitarian projects in Somalia

Somalia faces a protracted set of humanitarian crises, dating back to 1991 and civil wars following the end of Mohamed Siad Barre's regime. The country remains vulnerable to crises on multiple fronts including droughts, flooding, conflict and insecurity. In 2024, there were an estimated 6.9 million people requiring humanitarian assistance (OCHA, 2024b). Insecurity, including facing humanitarian actors, is heightened in areas of Somalia controlled by the insurgent organization Al-Shabaab (Barter and Sumlut, 2023).

Third-party monitoring has been operating in Somalia since at least the early 2010s when it became integrated into some funders’ multi-year programs, a response to large-scale corruption and aid diversion scandals around the 2011–2012 famine in Somalia. One interviewee explained third-party monitoring as a trade-off in relation to the MESH evaluation project:

‘DFID launched their first multi-year humanitarian-funded program in Somalia in 2013–14. The trade-off for that was we’ll provide this longer funding but there has to be a robust third-party monitoring process in place’.

Increased risk – tied to increased funding and longer-term programming in insecure contexts – was accompanied by increased donor oversight over implementation.

Third-party monitoring further expanded during the COVID-19 pandemic as donors had to rely even more on remote management (OCHA, 2021: 39). One interviewee from a large bilateral donor asserted that in Somalia

‘all of the donors now have their own specific monitoring systems in play, especially in multi-year funding. That is always coupled with a significant spend on third-party monitoring. … It's now a crowded field eight years since I’ve been doing this’.

There has been a proliferation of international and local actors in Somalia involved in collecting, analyzing, and reporting data for third-party monitoring. This has become an increasingly professionalized industry in Somalia including both international third-party monitoring firms and locally based partners.

Humanitarian donors have tended to contract international or foreign firms for large monitoring projects. These firms sub-contract local companies for specific activities and smaller budgets. Often, the international firms have business activities outside the humanitarian sector. For example, two of the early adopters of third-party monitoring in Somalia, DFID and BHA, both worked through firms based outside of Somalia. By 2015, DFID was working with four international third-party monitoring firms: Adam Smith International, Coffey International, Transtec/GDSI and IBTCI (International Business and Technical Consultants) (Integrity Research and Consultancy, 2015). USAID/BHA has also used third-party monitoring in Somalia as a remote management tool (Harrison, 2020; Integrity Research and Consultancy, 2015). U.S. Agency for International Development worked with IBTCI, headquartered in Washington DC, for a US$28.8 million third-party monitoring program in Somalia that ran from 2014 to 2021 (USAID, 2020).

Data generated through third-party monitoring is often done through a combination of methods, including surveys with aid recipients to assess if, and to what extent, delivery aligns with plans, as well as recipients’ needs and priorities. Usually, data collection and analysis are structured processes involving questionnaires. Mobile phones are used and are usually supplied to enumerators by the third-party monitoring firm. Monitoring and Evaluation for Somalia Humanitarian Program, for example, drew on household data of 2688 households and quantitative analyses, as well as some qualitative discussion (MESH, 2022). Questionnaires are pre-loaded using on survey-management software (e.g., KoBo Collect or Open Data Kit Collect). Once collection is complete, data is transferred to the third-party monitoring firm to clean and analyze. Mobile phones used for data collection are returned, and data is meant to be wiped (though some interviewees involved in data collection indicated they did not always immediately delete data). Data analysis, visualization and reporting do not usually involve data enumerators. Enumerators explained that data collected often, but not always, includes personally identifiable data. Questionnaire data is sometimes combined with GPS data and/or photographs where the capability exists, which are used to verify enumerator activity.

The data generated in third-party monitoring is not necessarily of huge volume, variety and velocity. Nonetheless, as will be discussed below, these data practices are framed by claims that third-party monitoring provides more data and more reliable data. This article considers the claims of what third-party monitoring data practices are imagined to achieve, reflecting on discrepancies between claims, and data practices and characteristics.

Data narratives around third-party monitoring in Somalia

Across interviewees, there was a consistent view that more data and quantified data were improvements over qualitative information in the sector. Interviewees from humanitarian agencies and donors were attuned to the difficulty of information flows from aid recipients, given they operated in high risk and insecure contexts, and made use of remote management practices. There was a relatively uncritical narrative about data in this context. One interviewee, working in humanitarian feedback, suggested that underpinning third-party monitoring is ‘this paradigm of “if we had all this data we’d be able to better solve these problems”’, what he saw as ‘a continuation of the tech-fix approach we’ve had for decades’. Another interviewee, from a large humanitarian implementing agency, commented ‘there's … a common consensus that you need a lot of disaggregated data and that should be a requirement for reporting’. Similarly, in third-party monitoring, a MESH vision statement claims:

‘Humanitarian action is being impacted by the ‘data revolution’. We can no longer expect any one expert, no matter how prolific and profound, to give an adequate analysis of the problems we face … even incomplete and anomalous data can provide signs that lead to ever more telling, compelling, and valid insights’ (MESH, 2024).

While there was a consensus that more data and quantified data provide for better information, narratives diverged about whether data through third-party monitoring firms was preferential over data from other actors, such as humanitarian agencies. Humanitarian agencies involved in implementation, in particular, were critical of risks of third-party monitoring, noting the firms’ distance from activities on the ground. The third-party monitoring firm could unintentionally compromise the security of delivery or relationships with local communities.

In contrast, donors, and, unsurprisingly, third-party monitoring firms, consistently suggested that third-party monitoring data improved humanitarian accountability by improving information flows between donors and targeted crisis-affected communities. A USAID-funded third-party monitoring firm self-described as ‘an independent, impartial, rigorous third-party monitoring system capable of regularly and accurately gathering, verifying, and reporting primary data on BHA-funded activities in Somalia’ (Integrity Global, 2022). Donors interviewed described third-party monitoring firms as more independent with ‘better’ data analytics than other options for monitoring and evaluation, such as self-assessment by implementing agencies. Third-party monitoring firms also were seen to understand the informational needs of donors better than alternative evaluators. One interviewee explained that third-party monitoring firms would provide ‘well washed’ data to their organization. In other words, the third-party monitoring firm derived insights from data for the donor that the donor could easily use in internal discussions (program manager, bilateral donor agency).

There was a willingness among the donors to trust the insights provided by the third-party monitoring firm. Trust was also necessary for donors, given they tended to operate remotely and could not triangulate data directly themselves. The basis of trust was linked to distance from humanitarian programming: third-party monitoring firms were not involved in implementation and often not even based in the communities of operation. Humanitarian donors consistently expressed trust in third-party monitoring firms to provide more impartial and accurate information than actors embedded in local communities and/or the humanitarian intervention.

Somewhat contradictory, their distance from the community and intervention was suggested to help the third-party firm to close the gap between humanitarian donors and affected communities, specifically by removing bias. Our donor interviewees articulated a desire to get as close as possible to the views of aid recipients, to better understand if and how their funded programming was having the desired effect in the target communities. However, they wanted to avoid relying on locally based firms or humanitarian implementing partners to mediate access to the views of target communities; these firms’ embeddedness in the intervention and/or its content might constrain what aid recipients shared, or what was reported back to the donor. One interviewee explained information might be tainted through coercion of aid recipients by local actors (senior consultant, bilateral donor agency). This concern motivated their trust in third-party monitoring firms as an alternative to humanitarian agencies and other actors who were anchored in the program and/or community (see also Sagmeister and Steets, 2016).

Alongside, donors assumed third-party monitoring firms were highly proficient at analyzing and interpreting data. This was supported in some cases by frequent communication between the third-party monitoring firm and donor, for example, regular weekly updates (coordinator and manager, bilateral donor agency). Also, some of third-party monitoring firms interviewed – though not all – explained how they provided assurance of their data management practices to donors, proactively sharing data practices and updating on data collection and deletion.

As a result, some interviewees from donor organizations saw the third-party monitoring firm as almost synonymous with the donor. One senior staff in a large bilateral donor likened third-party monitoring firms to an extension of the funding body itself. Expanding on this, they suggested,

‘I’m absolutely sold on the need for third-party monitoring … Our own ability to generate data is quite limited, so we need agreements, call center capacity, [and] monitors on the ground’ (team leader, bilateral donor agency).

Even while aligning closely with the third-party monitoring firm, donors were not naive to the potential for third-party monitoring firms to misuse their trust. One bilateral donor interviewed, after explaining their trust in the third-party monitoring firm, commented, ‘at the end of the day, who polices the monitors? Who polices the police? There will always be that eternal question’. Yet, relative to other options for evaluation, donors expressed a willingness to trust a third-party monitoring firm.

Looking across the ways that interviewees discussed the role of third-party monitoring firms, we find, in general, a critical orientation towards accountability in the humanitarian sector and a less critical view of data. For donors, there was a deterministic view that data was necessarily preferential over other forms of information gathering as a basis for knowing the realities of programming on the ground. Further, while normative views of the value of data through third-party monitoring firms varied, those funding third-party monitoring were ‘sold’ on its value. This was based on assumptions about firms’ distance from interventions, firms’ data analysis and interpretation skills, and their close communication with donors.

Data practices in third-party monitoring in Somalia

From here, we examined how the narratives around third-party monitoring firms translate into data practices in third-party monitoring, especially as it involves interactions with other actors in the sector for data collection, sharing and analysis.

We focus on data practices around data collection in the location of the humanitarian intervention. This emerges as a key point within data practices in which third-party monitoring firms seek to ensure distance and neutrality from the crisis context and community, while simultaneously relying on sub-contracted local actors. This poses a challenge to the narrative of third-party monitoring firms as distanced from the humanitarian intervention. As will be illustrated, third-party monitoring firms in Somalia aim to reclaim the objectivity of their data by removing traces of contextual or human influence. In this section, we discuss two implications of these practices.

First, the efforts to ensure data's objectivity produce a form of humanitarian ignorance about the data, how it is produced and its characteristics. This occurs through ‘dislocations from knowledge’ (see Cheesman, 2024) whereby the people and context for data collection are separated from the data, through precarious labor relations and remote analysis. Second, not only do practices produce an ignorance that sustains narratives of third-party monitoring firms and their data as more accurate due to distance, practices themselves accentuate power asymmetries. These asymmetries transfer risk to local Somali enumerators involved in data collection, while constraining their voice in interpretation of data. This ends up reproducing asymmetries between the Global North and South, as opposed to enabling information flows that empower the preferences of those affected by crisis.

These effects – humanitarian ignorance about data and power asymmetries – emerge through two features of data collection: 1) the way that distance is maintained from the crisis context; and 2) the nature and scope of surveillance over people involved.

‘Distant’ data analysis versus sub-contracted, in-person data creation

Most third-party monitoring firms' data practices can be conducted remotely, including analysis, data storage, visualization and reporting. However, data collection has continued to involve physical presence in the crisis-affected communities. Our interviewees explained that third-party monitoring firms collected data through in-person surveys and observations to ensure a more comprehensive view of practices on-the-ground. For example, one third-party monitor commented that remote data collection might miss input from female aid recipients without phones (female, third-party monitoring firm).

While emphasizing in-person data collection, third-party monitoring firms were limited by insecurity concerns (Integrity Research and Consultancy, 2015). This has contributed to a reliance on sub-contracting data collection to Somali-based firms and local enumerators, while activities that could be done remotely, such as analysis and reporting remained the responsibility of the third-party monitoring firm. Data collection, with its associated risks, was sub-contracted but data analysis remained with the external firm.

This results in a gap between data collection and data analysis: those involved in collecting (or creating) data, who have a greater understanding of the context for data generation, tend to have little insight into where data goes, how it is analyzed, and who or what it informs. Enumerators and local third-party monitoring staff interviewed expressed this uncertainty. They were usually restricted to data collection, under pressure from the lead third-party monitoring firm to follow data protection protocols. For example,

‘Enumerators are often not very well informed, they may not see the reports, they don’t anticipate the discussions they will have, etcetera. When we want to see the report, it is often difficult to get that information that you send back. [With] all of the deadlines that you have to meet and demands from the donor, often people who collect the data are forgotten’ (Somali head of Somali-based third-party monitoring firm).

Therefore, this interviewee recognized the omission of enumerators from analysis but noted it was potentially a feature of the monitoring project and its tight timelines, as opposed to intentional.

Both intentionally and unintentionally, there is a distance between data collection and data interpretation, as well as between sub-contracted Somali partners and the primary third-party monitoring firm. A hierarchy emerges, whereby the distanced, often Global North-based expert, has designed the study, and analyses and authors the findings, while the Global South partner co-creates data with the aid recipients and bears the associated risks, but is excluded from analysis and direct communication with donors. The concentration of planning, analysis and reporting within the lead third-party monitoring firm maintains the narrative of third-party monitoring firms’ distance. At the same time, it is contradicted by the realities of sub-contracted partners in data collection on the ground.

Surveillance measures to minimize human ‘bias’ and ‘error’ in data collection

In addition to preserving distance from the intervention where possible, third-party monitoring firms in Somalia sought to minimize the influence of data enumerators on data.

Donors identified a perceived trade-off between the value of in-person data collection in finding out realities on the ground, and an assumed bias associated with those involved in gathering data on the ground. One USAID report described community-based enumerators as ‘an advantage and a risk’ (USAID, 2021: 7). It weighed enumerators’ access against their assumed bias and potential for corruption, equating distance with objectivity.

Paralleling this concern, third-party monitoring firms in Somalia have exercised control and surveillance over enumerators before, during and after data collection. Surveillance has focused on protecting the objectivity of the data from enumerators’ subjectivity and potential error. At the onset, third-party monitoring projects usually require enumerators to undergo training on data protection, consent and the collective sectoral commitment to ‘do no harm’. Enumerators’ experiences of training varied, with some finding them insightful and others noting language barriers and a lack of training on key concerns, such as how to ensure quality data. During data collection, enumerators have been monitored digitally, including through photos and GPS readings to track where they have gathered data. Following data collection, data is supposed to be deleted from local enumerators’ devices, and devices returned to the third-party monitoring firm.

While enumerators were surveilled about the data they collect, our interviewees did not discuss surveillance by third-party monitoring firms to monitor security risks facing enumerators. Enumerators explained that sub-contracting agreements between third-party monitoring firms and local partners devolve responsibility for risk to the individual enumerator. Their work was often through short-term and casual contracts, which reduced the protections available to them. One female Somali enumerator explained:

‘I am not a full-time employee; I have no rights nor protection. I don’t have an ID … The terms in the contract are short term and only define the days of work and the pay I will receive. There are no terms that will safeguard me if something happens to me in the field, I get kidnapped, and so on’.

Another interviewee, also a female Somali enumerator, stated:

‘If something happens to you, especially, when operating in Mogadishu, there is no protection and no guiding measures. Roads are sometimes blocked while we are in the field, and different dialects are used in the IDP [internally displaced persons] camps. It is our own problem and we ought to solve’.

Evident in these quotations, uneven surveillance of enumerators fed a perception that the third-party monitoring firm was not concerned about the person involved in creating data. Several Somali enumerators interviewed described how safety and compliance with local social norms were their individual responsibility. They adapted their actions to build trust and comply with local norms. For example, noting local suspicions about face masks during the COVID-19 pandemic, one enumerator chose not to comply with the third-party monitoring firms’ COVID-19 mask requirements. In another case, an enumerator explained they refrained from recording information on aid diversion practices out of concern for their personal safety.

From an accountability perspective, these dynamics have several implications. First, they dehumanize data collection, focusing on the data, but obscuring and neglecting the people involved. This transfer of risk and risk mitigation to local sub-contracted enumerators reflects wider practices in insecure environments, including academic surveys (Davis and Wilfahrt, 2024), whereby the knowledge contributions of local partners are devalued in the process and publication of knowledge (Branch, 2018: 80–81). In this way, data collection practices reproduce a hierarchy between Somali actors and the (usually) foreign-based third-party monitoring firm, with the efforts taken to improve accountability through improved information flows feeding a historical binary between Global North and South (see Duffield, 2016; Madianou, 2019).

Second, the focus on people as a problem in data collection occluded what the data represents, and how it is tied to a particular context. Data enumerators interviewed shared multiple and varied ways that local context and interactions with interviewees influenced information sharing, and, through this, the data (co-)created by the interviewee and enumerator. Personal networks were identified as important for securing enumerator contracts, restricting who was involved in enumeration. During data collection, enumerators identified high levels of mistrust tied to their crisis-context, as well as ‘survey fatigue’, influencing what survey respondents discussed. Adaptations to manage security risks, identified above, indicate a dynamic process whereby managing security informs the degree of trust in sharing information. Enumerators also explained that crisis-affected communities are tired of responding to questionnaires by humanitarian agencies and donors, as they are surveyed for needs assessments and monitoring and evaluation by different agencies and related to different projects. They suggested questionnaire fatigue could even create resistance to the questionnaire. For example:

‘There is also a fatigue on the respondents’ point of view with almost nothing getting improved in their life. We face their wrath during interviews, and they are quite dissatisfied with the number of data they have to give with no substitute’ (male Somali enumerator).

Individuals from the crisis-affected communities could not, and did not, necessarily differentiate between data collected for humanitarian agencies or third-party monitoring firms. One Somali third-party monitoring staff stated, ‘Some of the challenges is, when you are talking to the beneficiary, they think you are from the implementing organization. They think if they share with you the real situation, that they lose their benefits’. This was further complicated as some of our interviewees explained they would conduct interviews for multiple actors and exercises (e.g., needs assessments as well as evaluations), not solely third-party monitoring firms. The face, then, of data collection to affected communities might be the same across different purposes.

Third-party monitoring firms’ concern for objective data contributed to a lack of awareness and interest in how immediate relationships around data (co-)creation and socio-political contexts are part of what data is. Their surveillance efforts focused on ensuring that enumerators go to the places they have been instructed to go, speak with the people identified and delete data when required. While there was some recognition in third-party monitoring guidance documents about security risks, this was a secondary concern. It was not discussed as a factor that might shape data itself, for example, contributing to what information is shared by affected communities, within which contexts, and what then is recorded and analyzed. This reduces data to something technical, neglecting how data is shaped through social and political relations. The ways that enumerators navigated insecurities, ensuring their safety and building trust with the survey respondents, shapes what information is shared, and what data is created, recorded and transferred for analysis. Yet, these characteristics of the data have been excluded from the third-party monitoring firm's view as it seeks to remove any traces of enumerator subjectivity.

Discussion

With the prevalent assumption amongst humanitarian donors and third-party monitoring firms that more data and new data analytics are necessarily good for humanitarian accountability, we find that third-party monitoring is in fact reproducing power asymmetries through particular areas of humanitarian ignorance.

Humanitarian donors contracted third-party monitoring firms to help improve information flows in order to better know and act in response to aid recipients’ experiences. As discussed above, interviewees were attuned to the difficulty of realizing accountability to donors and to crisis-affected communities. However, they were much less critical about data as a means to improving accountability by altering information flows.

The combination of a critical view of humanitarian activity and a less critical view of data has meant that third-party monitoring firms treat accountability as a technical issue that can be addressed through more current data, detached from contextual influence (similar to Sandvik et al., 2017). We found third-party monitoring firms have sought to limit the influence of people involved in data collection on data. This has included restricting data enumerators’ roles, as well as surveilling data collection processes. They have focused on where data is collected, when, by whom and to whom. They fail to consider data creation as shaped through interactions between people, with context and relationships part of what the data is. This selective knowing prevents a fuller engagement with the complex realities of how data is created, and the social, political and cultural dynamics that inform what is recorded as data. This is one element of what Fejerskov et al. (2024: 6) articulate as a ‘strategic ignorance’ utilized by the digital humanitarian sector, where purposefully not knowing certain information can be used to protect organizational liability or interests through a process of ‘rendering facts inert through sequestered knowledge’. Even in a process of data collection (or creation), ‘inconvenient data’ – in this case, the contextual and relational information provided by enumerators – can be effectively separated from the ‘reported knowledge’ readily quantified by digital humanitarianism and its monitoring processes.

The conditions for data enumerators, especially the nature of sub-contracts and devolution of risk, reinforce this ignorance about the realities of data as produced in context, further implicating accountability relations. Looking at a blockchain pilot, Cheesman (2024) found blind spots and erasures in knowledge through power asymmetries are one way that humanitarian ignorance is produced. Similarly, precarious subcontracts and hierarchical processes for data analysis and interpretation devalue and constrain the influence of Somali enumerators’ voice in producing information about humanitarian activities. We find that these conditions not only produce ignorance about the complexities and dynamism of data creation but also they reproduce power asymmetries, which value the expertise and lives of distant actors in the Global North and devalue the lives and knowledge of Somali partners involved in data creation with aid recipients (Fast, 2023; Madianou et al., 2016).

Donors imagine they have a more objective view of how their resources are used and shape crisis-affected people's experiences, but the interplay of data narratives and practices obscure how data relates to aid recipients’ experiences and perceptions. Donors’ critical orientation towards challenges of humanitarian accountability and a relatively uncritical orientation towards data have resulted in data practices that hide the complexities of what data represents. Through this, third-party monitoring reproduced power asymmetries between the Global North and Global South, devaluing and hiding the people and interactions through which data is created. This takes place even as some humanitarian actors, local Somali partners and enumerators interviewed argued for a contextualized view of data and the importance of being embedded in crisis-affected communities to build trust with aid recipients.

Conclusion

In our critical look into the data practices in third-party monitoring in the humanitarian sector in Somalia, we find that the entrance of external firms brings new challenges to accountability relations to donors and to aid recipients.

This takes place first, as data practices reproduce forms of ignorance about humanitarian activity and experiences of aid recipients, by considering data as technical and obscuring the dynamic role of people in data creation. Second, data practices reproduce power asymmetries between the Global North and Global South, devaluing the knowledge and lives of local actors.

With these findings, we do not suggest humanitarian donors are fully oblivious to critiques of data. Other studies have highlighted their awareness of the need for safeguarding against function creep, privacy breaches and potential harm to aid recipients (e.g., Diepeveen et al., 2022; Fast, 2022). Still, we do find a difference in how donors perceive data obtained through external firms versus information produced within the context of the humanitarian intervention, with external firms’ data seen to be more reflective of realities on the ground. Even as critiques of data have become well established in scholarship (Elish and boyd, 2018), we find less critical views of data continuing to feature within the context of humanitarian interventions.

This work expands on studies of humanitarian ignorance around digital technologies in the sector (Cheesman, 2024; Fejerskov et al., 2024). It shows how forms of knowing and non-knowing converge around monitoring and accountability, with ignorance about data contrasting with awareness of challenges of accountability in the sector. Our findings are also consistent with arguments that data and digital technology in the humanitarian sector constitute technocolonialism.

We argue that power asymmetries are reproduced around forms of knowledge and expertise, between and foreign actors, as well as extractive processes that place the power of interpreting data in the hands of third-party monitoring firms. When put alongside the experimentalism of piloting untested technologies in low regulatory contexts under the guise of innovative approaches (Jacobsen and Fast, 2019), a digitized and distant humanitarian sector becomes a vehicle for reproducing global power inequalities that devalue the lives of those within the Global South and formerly colonized world (Madianou, 2019).

Third-party monitoring was introduced to enable donors to hear from aid recipients about their experiences and better assess what their funded activity is doing, and to enable aid recipients to be heard and to inform humanitarian decision-making. Yet, even before considering what donors do with the information from monitoring and evaluation, we find challenges to accountability relations in data practices themselves, linked to the information intended to support accountability. The context and relational dimension of data creation is hidden and obscured, and an emergent hierarchy of knowledge devalues the expertise of local Somali actors involved in data creation.

Looking at directions in humanitarian policy, this study also points to continued contradictions in the aims of humanitarian donors and agencies. In addition to emphasizing accountability to crisis-affected people, the humanitarian sector has been increasingly focused on a localization agenda, whereby humanitarian agencies collectively aspire to better support and fund local and nationally based humanitarian organizations. This includes seeking to transfer more funding to national and local organizations, as well as address a lack of representation of local organizations in humanitarian coordination and assessment processes, and a perceived monopoly by international organizations on ‘objective’ knowledge production in crisis response (Barbelet et al., 2021: 9). Instead, in a pursuit of a perceived objectivity in interpreting humanitarian crisis responses, digitally enabled and distant monitoring and evaluation efforts risk contravening another key, though often overlooked, humanitarian principle of humanity: that of treating those in need with dignity and respect, and alleviate suffering. Expanding the gulf between the sources of data and those who use and profit from that data exacerbates a humanitarian component of a ‘digital extractivism’, whereby value from data created through humanitarian interventions does not go to affected communities and instead forms a valuable currency for aid actors to govern spaces and systems of relief (Iyer et al., 2021; Sandvik, 2023; Squire and Alozie, 2023).

Going forward, we suggest there is a key role for scholarship that not only critiques humanitarian data practices but also explores alternative data practices that are contextually informed and make visible local actors, experiences and relationships. For example, Walter and Suina (2018) argue that there are possibilities for linking quantitative methods to marginalized forms of knowledge production through self-determination in the entire process of knowledge production, including choices about data, data collection and instruments. Such contextually informed data practices contrast with the technical and distant approach to data in humanitarian third-party monitoring and evaluation. They create space to consider situated data practices that might be capable of producing different power dynamics and information flows, and potentially support changed relations of accountability.