Explaining sentiment shifts in UN system annual reporting: a longitudinal comparison of UNHCR,UNRWA and IOM

Abstract

Annual reports are a central element of international bureaucracies' accountability communication to member states and other stakeholders. Most UN system bureaucracies produce reports of significant length and detail. International agencies use these reports to draw attention to particular challenges or successes. Hitting the right tone with their diverse stakeholders is crucial to maintain continued support. UN agencies do so by employing differentiated sentiment-loaded language alongside factual reporting. We argue that agencies' operational focus, administrative structures and resource mobilization needs have a significant impact on how they use sentiment to communicate with different stakeholder groups. Drawing on a dictionary-based sentiment analysis of three text corpora of annual reports produced by three UN system agencies—UNRWA (reports published from 1951 to 2019), UNHCR (1953–2019) and IOM (2000–2019)—we show a general trend toward increased positive sentiment use across all three agencies, coinciding with a period of stronger donor orientation. At the same time, we find a more volatile and agency-specific use of negative sentiment in response to field-level challenges that are communicated to stakeholders in line with agencies' evolving mandates. Through a text-as-data perspective, this contribution enhances our comparative understanding of the diverse and context-dependent language of international bureaucracies.

Points for practitioners

Reading UN agency reporting, practitioners need to be aware of the constraints and incentives that international bureaucrats face—notably operational focus, administrative structures and resource needs—that drive tone differences across reports and over time.

Introduction

Regularly reporting back to member states and other stakeholders is a standard task of the bureaucracies of international organizations (IOs). In the United Nations (UN) system, most agencies produce substantive annual reports, which are made available to central stakeholders, such as member states, as part of regular accountability procedures. ¹ These annual reports present information about agencies’ activities and performance in the previous year and, as mostly public documents, are often read with keen interest by critics and supporters alike. As routine bureaucratic exercises focused on internal accountability, annual reports could be expected to be technocratic administrative documents written in a diplomatic and technical language. Their prime readership groups include member state diplomats, UN experts and specialized non-governmental organizations, which suggests that concerns for accuracy and performance-reporting should be more important than readability and public attention.

However, UN annual reports not only increasingly employ visual elements (Johnson, 2011), but frequently also involve the use of sentiment-loaded language (Patz et al., 2021). An agency may highlight its successes and positive contributions, underscoring how it “was able to rapidly and effectively deliver development and humanitarian assistance by drawing on its operational strengths”. Conversely, it may contrast this with a focus on the challenging environment it faces, such as “financial constraints [that] affected the delivery of emergency services […] triggering severe protests” (both quoted from §20 in UNRWA 2019, ² our emphases). This use of sentiment-loaded language in accountability reporting is puzzling from a public administration perspective as increased sentiment use increases public attention (cf. Stieglitz and Dang-Xuan 2013), which may increase scrutiny and responsibility attribution that IOs do not like (Louis and Maertens, 2021, Chapter 6). The central questions are thus why UN agencies use sentiment in their administrative annual reporting, whether there are common trends across UN agencies and what factors explain differences across agencies and over time.

While previous research on sentiment use by public administrations has focused on the effect of sentiment in press releases on media reporting (Duval et al., 2021) or sentiment congruence in the news room (Meyer-Gutbrod and Woolley, 2021), our argument follows recent discussions on the context-dependent use of sentiment by IOs. International organizations are found either to respond to stakeholder expectations (Breen et al., 2020) or to strategically increase sentiment to engage political stakeholders (Busch and Pelc, 2019). Because annual reporting is a routine exercise that allows year-to-year comparison for drafters and stakeholders, substantive shifts in sentiment use from one year to another can be considered conscious linguistic choices that UN agencies make with the aim of gaining or maintaining support. Sentiment use in IO bureaucracy reporting is expected to depend on, and vary with, differing or changing stakeholder audiences, not primarily to generate media reception.

We argue that sentiment variance can be explained by three key factors. The first is operational focus, which defines the sentiment level of the activities that agencies report on. Second, administrative structures affect the degree to which field-level and global stakeholders expect negative events to be reported on. Third, agencies' dependence on (non-permanent) resources increases their need to engage key audiences. Diverging from previous contributions (e.g. Breen et al., 2020; Busch and Pelc 2019; Duval et al., 2021), we argue that UN bureaucracies modulate positive and negative sentiment separately (Patz et al., 2021).

Empirically, we study sentiment variation in annual reporting by three UN system agencies: the United Nations Relief and Works Agency for Palestine Refugees in the Near East (UNRWA), the United Nations High Commissioner for Refugees (UNHCR) and the International Organization for Migration (IOM). All three operate in the same broad domain of global refugee and migration policy, but vary with regard to their core operational focus, administrative structures and resource dependencies across agencies and across time. Conducting a dictionary-based sentiment analysis (Taboada, 2016; Taboada et al., 2011) to detect both annual variation and broader baseline shifts in sentiment use, we observe significant differences in its use over time and across agencies, driven by organizational and operational differences of the three UN bureaucracies.

This paper contributes to existing public administration (PA) literature in two main ways. First, it speaks to broad debates in PA and organizational studies through the lens of international agencies. In particular, our theory is grounded in a conception of UN bureaucracies as organizations with multiple and diverse stakeholders (cf. Connolly and Hyndman 2013, 2017; Schillemans and Busuioc 2015) at administrative levels ranging from the global to the field level (Benz et al., 2017; Patz and Goetz, 2019). Second, this contribution builds on recent developments in text analysis in PA research (Liguori et al., 2018), following calls to include text-as-data in its methodological toolbox (Hollibaugh 2019; Pandey et al., 2017). By using annual reports—a type of administrative document that has been studied to assess public sector organizations' interaction with national audiences (Mack and Ryan, 2007; Steccolini, 2004) and to understand non-profit agencies' use of emotion (Langton and West, 2016)—we are able to combine existing knowledge on a well-researched type of administrative document with novel methods to gain insights into international agencies’ engagement with stakeholders.

Affective language use by international bureaucracies

The design, functions and substantive roles of international and UN bureaucracies have long been a focus of scholarly attention (Cox and Jacobson, 1973; Elmandjra, 1973; Ranshofen-Wertheimer, 1945). The two decades since Barnett and Finnemore's contribution (2004), however, have seen a surge of theorizing about and conceptualizing of the role and impact of IO bureaucracies and how they differ from national and subnational public administrations (Bauer et al., 2017; Eckhard and Ege, 2016; Stone and Moloney, 2019).

Two key differences from other public administrations stand out when studying international bureaucracies. First, IO bureaucracies operate in a distinct “complex principal–complex agent” environment (Patz and Goetz, 2019) where states act as collective but often non-homogeneous principals while simultaneously interacting with IO bureaucracies as multiple individual principals (cf. Lyne et al., 2006). Voluntary earmarked funding, a major financing mode in the UN system that gives donors near-complete influence over where their funds are spent, provides a good example of the latter (Graham, 2017).

Second, IO bureaucracies do not possess the same level of authority, legitimacy and resource security that is commonly assumed—although not guaranteed—in other public administrations. As a result, IO bureaucracies develop distinct administrative styles (Bayerlein et al., 2020), provide emotional labor (Nair, 2020) and develop strategies of attention-seeking (Jörgens et al., 2017; Kolleck et al., 2017), brokering (Saerbeck et al., 2020) or orchestration (Hickmann et al., 2021) to engage with and mobilize stakeholders or increase the impact of their expertise.

Given their fragile authority and resources, perceptions of credibility and legitimacy of IOs by their stakeholders—namely government officials (Herold, 2019), diplomats sitting in executive bodies (Elsig, 2011), (international) non-governmental organizations (Tallberg et al., 2018), intermediary organizations (Hickmann et al., 2021), agency staff and beneficiaries—are crucial for IOs' ability to perform the tasks assigned to them through mandates or other arrangements (e.g. project funding). International bureaucrats who are perceived to be biased may see their expert authority undermined (Busch and Liese, 2017; Heinzel et al., 2020). Criticism from non-governmental organizations and national governments can also lead to a legitimacy loss (Dellmuth and Tallberg, 2021).

Thus, IO bureaucracies must pay particular attention to how their communication is perceived by diverse stakeholders at both the global and the field levels. Their performance is regularly evaluated, both internally and externally (Eckhard and Jankauskas, 2019; Lall 2017), meaning that reporting to these stakeholders in ways that address their concerns is crucial. One way that enables IO bureaucracies to address diverse stakeholders is embedding affect into their reporting, linking operational facts with sentiment that matches stakeholder expectations. This strategy is by no means unique to the international level (Song and Meier, 2018). For example, charities that face multiple stakeholder audiences use variation in “emotive disclosures” in their annual reporting to increase trust in their operations (Connolly and Hyndman, 2013, 2017; Langton and West, 2016).

Sentiment use in IOs is higher in politically salient domains relative to more technical ones to increase stakeholder engagement with agency output (Busch and Pelc, 2019). Thus, we expect reporting on operations in domains such as conflict resolution or humanitarian aid to have more sentiment-loaded language attached to it (negative: “war”, “conflict”, etc.; positive: “support”, “protect”, etc.) than reporting in more technical domains, such as patent regulation or standard-setting. In politically salient domains, agencies should generally use more affectively explicit language because states as primary stakeholders want to be alerted of politically important developments while agencies' beneficiaries and field staff expect their situations to be effectively communicated to donors and the international community.

Hypothesis 1: International agencies with operational focus in politically salient domains use more sentiment in their annual reporting than agencies in technical domains.

At the same time, IO bureaucracies cannot simply maximize sentiment use and turn administrative reporting entirely into political discourse as this would increase critical scrutiny by diverse stakeholders (Patz et al., 2021). Thus, IO bureaucracies need to strategically and selectively modulate their use of positive and negative sentiment to speak to their different stakeholders. To increase the targeted sentiment “punch” of their reporting, agencies can choose to increase either the negativity and positivity of their reporting, increase one while decreasing the other or increase/decrease both. Which modulation is chosen is probably influenced by IOs' operational environments and driven by their desire to respond to concerns of multiple stakeholders.

Negative sentiment shifts are expected to be strongly driven by field-level events and crises, especially in IOs that are more in touch with local stakeholders and beneficiary populations, as they are more immediately affected by negative events (e.g. humanitarian crises) and better equipped to perceive and reflect on field-level sentiment. They also have an interest in advocating for negatively affected beneficiaries and staff to global-level stakeholders as part of their resource mobilization strategies (Thorvaldsdottir et al., 2021). However, this is conditional on where agencies interact with key stakeholders (Elsig, 2011). Where accountability to global level stakeholders matters less (e.g. for financing, see H3) than reporting to field-level stakeholders, global (annual) reporting should have lower levels of negative sentiment and focus on positive success reporting (H3).

Hypothesis 2: International agencies with a focus on field level operations and who are dependent on global-level stakeholders (for finances or legitimacy) make greater use of negative sentiment in their global annual reporting than agencies without operational field-level presence or with primarily field-level stakeholders.

Positive sentiment, in turn, has quite different determinants. As IO bureaucracies increasingly compete for resources, they need to signal performance (Lall, 2017), accountability and project success to stakeholders that can provide material or ideational resources, all of which are likely to lead to increased use of positive sentiment. The trend toward earmarking of voluntary funding in the UN system (Graham, 2017; Patz and Goetz, 2019) requires even greater highlighting of success and effective mandate delivery, so we expect positivity to become increasingly pronounced in (annual) reporting over time.

Hypothesis 3: International agencies depending on precarious resources and who compete for funding with other agencies will use more positive sentiment in their annual reporting than agencies with stable and uncontested resources.

Overall, while annual reports are not the only way in which stakeholders receive information about IO operations (cf. Mack and Ryan 2007), they are the most comprehensive agency communication. In annual reports, IO bureaucracies need to address all relevant stakeholders, so increased sentiment use is driven by diverse internal and external pressures to reflect stakeholder concerns at the global and field levels. Significant sentiment shifts in annual reporting can then become indicators for major shifts in the operational or stakeholder environment of international bureaucracies.

Case selection and empirical expectations: UNHCR, UNRWA and IOM

To empirically examine the hypotheses formulated above, we compare sentiment use across three UN agencies: UNCHR, UNRWA and IOM. The selection of these three agencies allows us to hold constant features that are specific to the domain of global refugee policy while retaining variation in other dimensions important for our analysis, namely organizational focus (H1), administrative structures (H2) and resource dependencies (H3). Supplemental Table A1 provides summary details on the three organizations and links the three dimensions to specific empirical expectations for each agency. Owing to the differing time spans of publicly available reports for the three agencies, we only compare the more recent IOM reports with the newer reports from the other two agencies. We do not make inferences with regard to older IOM reports or cross-agency comparisons between old and new reports.

All three organizations issue annual reports on their activities in the previous administrative year. The reports usually come out well into the subsequent year. Sentiment contained in these reports is based on at least several months' hindsight and is thus strategically employed. The reports are publicly available, but are primarily administrative documents published for a diplomatic context focused on core stakeholders, not the general public. The length, of the reports by UNHCR and UNRWA track each other fairly well until the mid-1990s, when the UNRWA reports become significantly longer. Subsequent to a length limit imposed in the mid-2000s, the two organizations have converged in length again. IOM, however, tends to have slightly longer reports than the other two agencies (see supplemental Annex II for details).

Annual reports in all three agencies are built on a paragraph-oriented narrative describing activities and performance alongside relatively neutral visual elements such as charts and tables (which are removed for the sentiment analysis below). UNRWA and UNHCR are required by statute or UNGA resolution to submit this annual report to the UN General Assembly (UNGA Resolutions A/RES/302, A/RES/58/153). Documents and interviews reveal that IOM and UNRWA describe their annual reports as tools for reporting highlights or a broad overview of activities (IOM 2000, §3; IOM 2017, §1; Interview UNRWA Officials #4, #16).

This requires reporting on broad dynamics relevant to the UN administrative system and to member state diplomats, in a language that is sensitive to their concerns. Interviews with UNRWA officials confirm that the writing starts at the field and policy department level and works its way up through the organizational hierarchy with report consolidation by the planning department and language filtering by the legal department and executive office (Interview UNRWA Officials #4, #16). IOM annual reports suggest a similar mechanism, where the different field offices provide material through a questionnaire that is then merged and edited into a final version (IOM 2017, §1; IOM 2018, §1). In all cases, available evidence and the length of the documents suggest that annual reports are collective and organization-wide products and not the work of single author. Sentiment variation is thus largely independent of individual officials' writing styles.

Specific expectations with regard to variation in sentiment use can be gleaned from analysing the organizational variation described in supplemental Table A1. Variation in operational specialization suggests that UNHCR and UNRWA should use more overall sentiment, owing to their humanitarian mandates, than IOM with its historical focus on migrants and lack of formal mandate (H1). Given IOM's strong projectization and dependence on field-level fundraising without strong reliance on global level stakeholders (Hall 2016: 89–90; Patz and Thorvaldsdottir 2020), negative events should be reflected less in the sentiment of global IOM reporting than in that of either UNHCR or UNRWA (H2). Conversely, the close proximity of UNRWA headquarters (in Amman and (East) Jerusalem) to its field operations and beneficiary population suggests that it should be more sensitive to negative events than either of the other two organizations. The clear division of mandate between UNHCR and UNRWA also means that major crises affecting the field outside the Near East should not affect UNRWA reporting, while conflicts and other negative events affecting Palestine refugees should not be mirrored in UNHCR reports (implicit in H2).

Finally, with increasing resource competition and professionalization of resource mobilization (JIU 2007, 2014), we expect all three agencies to increase their positive sentiment in this period to highlight their successes to donors (H3). As UNHCR and IOM compete more directly (Patz and Thorvaldsdottir, 2020), trends in positive sentiment use in these two agencies should be more highly correlated than that of either organization with UNRWA.

Analysis: differentiated sentiment baselines, event-driven peaks and increased positivity in the three agencies

To identify sentiment in UNHCR, IOM and UNRWA's reporting, we use the Lexicoder Sentiment Dictionary (Benoit et al., 2018; Young and Soroka, 2012). ³ This dictionary has been used both in prior IO research (Busch and Pelc, 2019) and on national public administration documents (Duval et al., 2021). We analyze sentiment shares at the document level over the entire period for which reports are available for the three agencies (1951–2019 for UNRWA, 1953–2019 for UNHCR, 2000–2019 for IOM). To prevent our results from being driven by shifts in document length, we report negative and positive sentiment normalized as shares of the total word count of the document and the overall sentiment as a sum of the two shares (see also supplemental Annexex II and III).

In addition to different short-term responses in sentiment use by the three agencies to events and crises, we also expect the agencies to have different baseline levels of sentiment at different points in their history in line with major dynamics theorized in H1–H3. To test this, we conduct a change-point analysis with binary segmentation (Lansdall-Welfare et al., 2016; Scott and Knott, 1974). ⁴ This technique allows us to identify baselines shifts in sentiment use that cannot be explained by year-to-year volatility. To control for the possibility that temporal shifts in sentiment are merely shifts in linguistic style, for example to make annual reports more accessible to the media or the general public, we compare their difficulty level over time using the Flesch readability score (Flesch, 1948). We find that reports tend to get more, rather than less, complicated over time (supplemental Annex II). Thus, there is no obvious reason to believe that sentiment changes are driven by a more journalistic communication style in the reports.

A comparative look at the evolution of overall sentiment in the three agencies (Figure 1, left panel) shows a trend toward its increasing use in annual reports. While around 7% of the words in the earliest reports from UNHCR and UNRWA (from the early 1950s) are sentiment loaded, that share increases to around 10% by 2019. All three agencies share a peak in overall sentiment in 2016, that is when the Syria refugee crisis reached Western Europe, with UNHCR reporting the highest overall sentiment across all three agencies, at 12%. However, the paths to higher sentiment baselines differ across the agencies. While UNHCR has incrementally increased sentiment in its reporting over time, UNRWA was fairly stable until the mid-1980s when it became more volatile (Figure 1, right panel). Interestingly, IOM significantly increases its sentiment use once it joins the UN system in 2016 (Figure 1, right panel).

Figure 1.

Evolution of the share of overall sentiment use in agencies' annual reports (left side, time series data; right side, changepoints with means).

Contrary to expectations (H1), there is no strong indication that the operational focus of the agencies—UNHCR's and UNRWA's humanitarian mandates and IOM's more logistical orientation—drives overall differences as sentiment levels are inconsistent over time. UNHCR increased its use of sentiment earlier than the other two agencies, whose overall sentiment levels are similar until the early 2000s, despite differences in operations. The notable increase in sentiment use by IOM after joining the UN system—driven by an increase in positive terminology—could, however, indicate pressures from new global stakeholders for a more humanitarian orientation, leading to increased convergence with the other two agencies, evidenced, for example, by the launch of IOM's inaugural “Principles for Humanitarian Action” in 2016 (IOM 2017, §28). The change could also reflect the more competitive global resource environment of the UN system and the consequent need to impress donors (H3). As a result, IOM changed the structure of its annual report “to show the breadth and depth of IOM's work during the year in a more results-oriented way” and “to provide a better representation of … what the Organization has achieved” (IOM 2016, §1).

Negative sentiment shifts in UNRWA, UNHCR and IOM

A differentiated analysis of positive and negative sentiment in the three agencies underscores the variation in the environments in which they operate. First, IOM has a noticeably lower share of negative sentiment than UNHCR and UNRWA and less pronounced variation in negative sentiment baselines (Figure 2, left panel). This fits our expectations given IOM's projectized and heavily decentralized administrative structures (H2). Indeed, country-level summaries in IOM's reports tend to focus on short project results rather than discussing the difficulties of staff and beneficiaries, especially as compared with UNHCR or UNRWA (see examples below). A notable exception is the negativity peak in the 2016 report, which suggests that the global importance of the Syria crisis to key global stakeholders and also IOM's politically sensitive involvement in cross-border operations from Turkey to Syria ⁵ had a significant impact on the agency's reporting (H1 and H2).

Figure 2.

Evolution of the share of negative sentiment use in agencies' annual reports (left side, time series data; right side, changepoints with means).

In contrast, negative sentiment in UNHCR and UNRWA is both higher and more volatile. Both started off at fairly similar levels in the 1950s but the impact of the 1967 Six-Day-War and the resulting displacement of Palestine refugees led to an increase in negative sentiment in UNRWA reporting that lasted for about two decades. Negativity in UNRWA peaked in the late 1980s and early 1990s when the First Intifada and the effects of the Gulf War massively challenged its field operations. With the high hopes for the Oslo peace process in the early 1990s and reduction of hostilities, negativity temporarily decreased for UNRWA but then increased again with the 2007 conflict and blockade of Gaza and continued toward the Syria war that heavily affected Palestine refugees and UNRWA field operations.

The close tracking of negative sentiment changepoints (Figure 2, right panel) with regional conflicts that create important challenges to mandate delivery while also negatively affecting key stakeholders—Palestine refugees, UNRWA staff and host countries—fits our expectation that a field-based agency with strong links between the field, its headquarters and the mandate-giving UN bodies in New York will communicate local challenges in its global reporting. These shifts reflect not only singular events but also prolonged periods of field-level challenges that are reported to global audiences as bureaucratic advocacy vis-à-vis diverse stakeholders. As UNRWA described its role in 2007:

UNRWA is a global advocate for the protection and care of Palestine refugees. In circumstances of humanitarian crisis and armed conflict, the Agency's emergency interventions, and indeed its presence, serve as tangible symbols of the international community's concern.

(UNRWA 2007, §4)

UNHCR's negative sentiment baseline increases gradually over time, short-term modulations notwithstanding (Figure 2, right panel). This mirrors field-level challenges generated by a steady increase in global refugee numbers and the growing field presence of UNHCR around the globe. The combination of this field presence and a global-level mandate means that large-scale crises, such as the Afghanistan and the “boat people” refugee crises in the late 1970s, the parallel crises in the Great Lakes region of East Africa and the Western Balkans in the early 1990s, and the Syria crisis in the mid-2010s, are clearly reflected in negativity increases in the reports (supporting H2). In addition, there are periods, such as in the early 2000s, when multiple concurrent challenges affect refugees and UNHCR staff alike, resulting in overall negativity increases even in the absence of a large-scale crisis:

Chronic instability in many parts of Africa forced hundreds of thousands of people to leave their homes […]. Conflict and civil strife also continued unabated in South-West and South-East Asia, the Balkans and Caucasus regions, and South America. Such situations were characterized by a lack of security for vulnerable civilian populations and humanitarian staff. Restricted access limited UNHCR's ability to protect and provide relief to many affected populations and impacted UNHCR's own security. The latter became brutally apparent when three UNHCR staff in West Timor, Indonesia, and another in Guinea were cruelly murdered while carrying out their duties.

(UNHCR 2001, §1; negative sentiment terms from the Lexicoder Sentiment Dictionary underlined)

While this paragraph contains factual descriptions that in themselves include (negative) sentiment (H1), the agency also uses negativity to underscore operational challenges that affect beneficiaries and staff, and that undermine its mandate (H2). In contrast, IOM, which is without such a mandate and lacks strong ties between the field and the global level, chooses to open its report in the same year with organizational successes (H2):

The principal overall trend for IOM in 2000, as you will see from this report, was growth. Growth in membership, growth in operational funding levels, growth in the numbers of offices and projects, and growth in the types of services being requested.

(IOM 2001, §3)

As noted above, however, the Syria crisis has left an indelible mark on all three agencies. None of them has included more negative sentiment in their reporting than in 2016, demonstrating the significant impact of the crisis on the agencies, whose operations scaled up significantly in response—in part owing to major member-state expectations—and who required emergency funding in support of these efforts.

Positive sentiment shifts in UNRWA, UNHCR and IOM

All three agencies have consistently moved to higher positive sentiment baselines (Figure 3, right panel), especially in the 2000s (UNHCR and UNRWA) and after 2016 (IOM). This fits our expectations based on resource dependencies and the increasing role of earmarked voluntary contributions (H3). In line with the link between resourcing and positive sentiment, UNHCR and UNRWA both had an initial positivity peak during World Refugee Year in 1959–1960, a year that brought additional resources to both agencies. However, we also observe substantial cross-agency variation in positive sentiment (Figure 3, left panel). First, UNHCR has had a higher positive sentiment share than UNRWA over almost the entire history of both agencies, which could indicate greater resource-mobilization needs by UNHCR. For UNRWA, shifts in donor community and expectations have meant that it has had to increase fundraising efforts since the mid-2000s (Patz et al., 2021).

Figure 3.

Evolution of the share of positive sentiment use in the three agencies (left side, time series data; right side, changepoints with means).

Joining the UN system in 2016 appears to be a major driver of IOM's increased use of positive sentiment (Figure 3, right panel). Background discussions with IOM regional staff (June 2018) suggest that this is due to the agency's more direct competition for funding with other UN system agencies. As discussed above, joining and reporting within the UN system may also come with increased expectations by major stakeholders to use “humanitarian” language (e.g. the use of the humanitarian term “protection” increases significantly between 2000 and 2019). More generally, IOM faces “increased demands … for information” (IOM 2017, §11) from stakeholders. Hence, the entry of the agency into the UN system places it in a new stakeholder environment in which it becomes paramount to demonstrate strong performance as the new “UN Migration Agency”, and the agency opts for more positive language in its global reporting compared with the more technical summary of central activities in the years prior (H1 and H3).

Conclusions

In this contribution, we argue that annual reports are an important tool for UN agencies to communicate with key stakeholders, such as member state principals, donors, agency staff and beneficiary populations. Empirically, we demonstrate significant and meaningful variation in the way that the three agencies—UNRWA, UNHCR and IOM—interact with their environments and employ sentiment to engage with their stakeholders. We find an increase in overall sentiment use over time in all three agencies. This trend is primarily driven by increased positive sentiment, probably as a response to competition for donor support but also, in the case of IOM, as a more pronounced humanitarian orientation after joining the UN system.

In addition, we find that baseline shifts in negative sentiment appear to reflect the distinct and varying operational challenges of the three IO bureaucracies in the field, conditional on their administrative structures. UNRWA and UNHCR, as field-based humanitarian organizations with strong global mandates and links between the field and headquarters, highlight negative events to their primary accountability audience, namely member states, to demonstrate their mandated advocacy for affected populations but also to ensure that global stakeholders understand the challenges of the agencies and provide continued support. Negative signaling is relatively weaker in IOM, which lacks a convention-based mandate. Lastly, we observe that events such as the World Refugee Year or the Syrian refugee crisis that affected operations of multiple agencies led to a simultaneous increase in positive or negative sentiment use. This underscores that UN bureaucracies are part of a common system and can be affected by the same trends and events.

These findings have implications for our understanding of public agencies' use of sentiment, for our understanding of accountability reporting by IO bureaucracies and for the UN system. First, sentiment use is not purely a function of text genre or type of speaker (Gray and Baturo, 2021), and sentiment may be increased without the aim of increasing message diffusion (Duval et al., 2021; Stieglitz and Dang-Xuan, 2013). However, increased sentiment use in response to stakeholder expectation also makes these documents more political (Patz et al., 2021), which may increase their impact on public discussions (cf. Busch and Pelc 2019). Second, accountability reporting varies even between international bureaucracies that are active in the same broad policy domain within the UN system and within the same text genre, providing increased evidence for the diversity of IO bureaucracies (e.g. Bayerlein et al., 2020 on their administrative styles). Third, studying sentiment in multiple UN agencies over extended time periods has provided indicators for events or trends with cross-agency or system-wide relevance. Future research using sentiment analysis could thus enhance our understanding of the UN as a system of independent but interconnected IO bureaucracies.

The dictionary-based sentiment analysis employed here at the document-level also has limitations: while the plausibility of the hypotheses was demonstrated, alternative methods of measurement (different dictionaries, supervised learning) and a focus on sentence or paragraph levels may reveal more complex micro-level dynamics than hypothesized. In addition, future research could examine more directly the impact of sentiment use on different stakeholders. While the effect of sentiment in administrative communication on media reporting has already been explored (Duval et al., 2021), experimental studies could identify whether and how IO bureaucracy stakeholders respond to shifts in sentiment.

Supplemental Material

sj-docx-1-ras-10.1177_00208523211029804 - Supplemental material for Explaining sentiment shifts in UN system annual reporting: a longitudinal comparison of UNHCR, UNRWA and IOM

Supplemental material, sj-docx-1-ras-10.1177_00208523211029804 for Explaining sentiment shifts in UN system annual reporting: a longitudinal comparison of UNHCR, UNRWA and IOM by Svanhildur Thorvaldsdottir and Ronny Patz in International Review of Administrative Sciences