Abstract
Advocacy evaluation has emerged in the past 20 years as a specialized area of evaluation practice. We offer a review of existing peer-reviewed literature and draw attention to the scarcity of scholarly work on human rights advocacy evaluation in the Global South. The lack of published material in this area is concerning, given the urgent need for human rights advocacy in the Global South and the difficulties of conducting advocacy in contexts in which fundamental human rights are often poorly protected. Based on the review of the literature and our professional experiences in human rights advocacy evaluation in the Global South, we identify themes in the literature that are especially salient in the Global South and warrant more attention. We also offer critical reflections on content areas not addressed in the existing literature and conclude with suggestions as to how activists, evaluators, and other stakeholders can contribute to the development of a field of practice that is responsive to the global challenge of advocacy evaluation.
Literature on advocacy evaluation predominantly addresses and originates from high resource and, more specifically, North American contexts (Gill & Freedman, 2014; Harvard Family Research Project, 2007; Mansfield, 2010; see Glass, 2017). There is an evident lack of research on advocacy evaluation from the Global South. 1 At the time of this review, there was one published literature review on advocacy evaluation (Glass, 2017). While Glass’s review is reflective of the field and provides a comprehensive summary of the past 10 years of research, it speaks almost exclusively to the Global North and well-resourced evaluation audiences. In that same year, Gardner and Brindis (2017) published a book that offers a comprehensive review of advocacy and policy evaluation emphasizing use in practice and accessible resources. This text was and continues to be essential to the advancement and consolidation of the field of advocacy evaluation. Still, the theories, frameworks, and recommendations put forth in this text reflect the ever-growing body of the literature coming from the Global North. In regard to the Global South, this book addresses human rights advocacy evaluation and offers case examples from the Global South; nonetheless, the authors are transparent that these case examples involved external evaluators and consultants from the Global North. There remains a gap in the existing literature on advocacy evaluation in the Global South.
The present article provides a literature review of peer-reviewed publications on human rights advocacy evaluation from the past 20 years and highlights findings and gaps that are particularly relevant to the Global South. The current scholarly literature provides guidance on how to do advocacy evaluation in the Global North (often relatively safe, stable, resourced, predictable, and transparent contexts). This article extends additional guidance to donors, evaluators, and activists 2 working in the Global South.
Background: The Development of the Field of Advocacy Evaluation
The field of advocacy evaluation emerged in response to a gap between traditional program evaluation and the challenges of evaluating advocacy initiatives (Devlin-Foltz et al., 2012). Researchers offer varying definitions of advocacy, but the following is salient and concise, Advocacy refers to the process by which individuals or groups increase support for, recommend, influence, or promote a private, local, state, or federal decision or policy. (Millstein & Sallis, 2011, p. 468)
In terms of the history of the field, the mid-1990s into the early 2000s marked the first attempts at establishing systematic approaches to advocacy evaluation (Mansfield, 2010; see Glass, 2017). Some of the first university-based human rights advocacy impact studies emerged in the mid-1990s with a focus in Latin America; the authors of these studies were optimistic about having an observable impact (Hafner-Burton & Ron, 2009). However, demonstrating impact has proven quite challenging. Evaluation practitioners started contributing to advocacy evaluation in the mid-2000s (Glass, 2017). Authors locate the majority of the development of models, tools, and frameworks for advocacy evaluation in the United States and Canada (Gill & Freedman, 2014; Harvard Family Research Project, 2007; Mansfield, 2010; see Glass, 2017). The Blueprint Research and Design’s (Guthrie et al., 2005) publication, The Challenge of Assessing Policy and Advocacy Activities, produced for The California Endowment, was one of the first publications to attempt systematic field development in advocacy evaluation (Morariu & Brennan, 2009). Then, in 2007, the American Evaluation Association launched the Advocacy and Policy Change topical interest group, and the Harvard Family Research Project dedicated an issue of the evaluation exchange to advocacy evaluation (Devlin-Foltz et al., 2012; Glass, 2017). In 2017, Gardner and Brindis published an authoritative book on advocacy and policy evaluation, which has become a mainstay in the field. Reflecting this growth and consolidation of the field, authors have noted the “entry of many new actors from traditional evaluation firms, academia, strategic communications organizations, and government relations firms, among others” (Devlin-Foltz et al., 2012, p. 583) into advocacy evaluation; something they consider a promising development, bringing expertise, competition, and better resourcing.
Despite meaningful growth in advocacy evaluation, the field remains limited. A review of advocacy organizations has found a “lack of systematic, rational evaluation and measurement of the effectiveness of advocacy” (Almog-Bar & Schmid, 2014, p. 15). Most organizations do not have written plans for their advocacy work, much less evaluation plans (Almog-Bar & Schmid, 2014; Coates & David, 2002; wHarvard Family Research Network, 2008; Mansfield, 2010). While there is a need for work and publications in advocacy evaluation generally, this literature review suggests that there is an even more substantial gap in the Global South.
This article offers a critical literature review of peer-reviewed publications on human rights advocacy evaluation. This type of review offers an overview of the existing peer-reviewed literature and then brings a critical lens to that body of work to suggest future directions for advancing the field of study under review (Grant & Booth, 2009; Paré et al., 2015). The aim of this critical literature review is to assess the extent to which work from the Global South is represented in peer-reviewed publications on human rights advocacy evaluation. To the extent that the Global South is represented, we summarize the unique content addressed in those publications. Additionally, we provide a succinct overview of the full literature review in order to offer critical reflections on how current literature does and does not yet reflect or serve the needs of human rights advocacy evaluation in the Global South.
We emphasize human rights advocacy evaluation in this critical review of the literature. While this limits the range of literature included, it remains a highly inclusive approach. Human rights are operationalized as rights belonging to all human beings. In accordance with the Universal Declaration of Human Rights, these rights include the following: civil rights (to life, liberty, security, privacy, freedom of movement, and freedom from torture and more), rule of law rights (to equal protection under the law, remedy for violation of rights, and more), political expression rights (to assemble, participate in government and elections, and more), economic and social rights (to standards to physical and mental health, levels of education, join unions, and more), and right of communities (to self-determination, protection of minorities, and more; United Nations, 1948). Indeed, this literature search included articles on a range of topics, such as environmental justice, health equity, and education advocacy, while also identifying key literature in advocacy evaluation (e.g., Glass, 2017). We elected to focus on human rights advocacy because it is especially relevant to the advocacy needs and campaigns in the Global South, where it is often more difficult to assure and protect fundamental human rights (Ron et al., 2017).
For a critical review of the literature, it is important for readers to understand the professional experiences and subjectivities of the authors offering their critical perspectives. Three of the authors are researchers at a human rights organization which is headquartered in the United States, has offices in Jordan, and sustained programming in Ethiopia, Kenya, and Uganda. Three of the authors reside in the United States and a fourth in South Africa. Between them, the authors have conducted an evaluation and provided training throughout Africa, Eastern Europe, the Middle East, South Asia, South East Asia, and the United States. All authors have professional experience in human rights advocacy evaluation, including the development and a pilot study of a human rights advocacy evaluation tool kit designed for use by activists in the Global South.
Additionally, one of the authors worked as a human rights activist in the Global South. All four authors are writing from what we consider to be a Global North position of power as White evaluators with advanced academic degrees and within well-resourced U.S. institutions. The range of personal and professional experience, in the Global North and Global South, equips the authors to provide informed reflections on the existing literature and guidance to activists, funders, and evaluators working in or supporting work in the Global South. However, the privileged Global North perspective of the authors also undoubtedly limits their perspective on these issues in crucial ways.
Through this critical review of the literature, we advance the following three arguments.
First, the literature review revealed a lack of representation of the Global South in peer-reviewed publications on human rights advocacy evaluation. There is also a general lack of acknowledgment of this gap within this body of work. The present article offers the simple but essential recognition that the Global South is not adequately represented in the existing peer-reviewed literature on human rights advocacy evaluation. Inevitably, the relatively narrow set of voices contributing to debates about the theory and practice of advocacy evaluation limits the depth and generative capacity of this field.
Second, as a growing field, advocacy evaluation has several gaps, limitations, and calls for more research. This is well established in the existing literature. There are, however, advocacy evaluation considerations that are specific to, and amplified in, the Global South. Because the existing peer-reviewed literature comes predominantly from the Global North, the findings and recommendations are often poorly fit to the needs and conditions in the Global South. This is particularly important since this type of scholarly knowledge influences ways of operating in the field of human rights advocacy evaluation through power-laden structural and social mechanisms (e.g., funder requirements, external evaluation, internalized standards of practice). We offer a critical appraisal of likely needs and considerations based on our own professional experiences in the Global South that are not addressed sufficiently in the existing literature.
Third, the lack of research from the Global South leaves several currently influential actors in the field—notably funders and external evaluators from the Global North—without essential information about effective, ineffective, or even detrimental evaluation approaches in the context of the Global South. As such, activists in the Global South are not equipped, or allowed by Global North funding institutions, to use evaluation resources that adequately support the advancement of human rights. The scholarly literature on human rights advocacy evaluation must keep pace with the on-the-ground developments in human rights advocacy in the Global South. To this end, we offer concrete actions that funders, evaluators, and other stakeholders (from the Global South and the Global North) can take to support the inclusion of research and evaluation from the Global South in the peer-reviewed scholarship.
Literature Search Strategy
This critical review includes peer-reviewed literature between 1997 and 2017 on the evaluation of human rights advocacy initiatives. We focus on peer-reviewed literature because of the important role this body of work plays in guiding practice, particularly through the mechanism of funder requirements and external evaluation approaches. The noted 20-year time frame was selected because it encompasses the earliest systematic published work in advocacy evaluation through the most recent publications at the time of the search.
The search emphasized work done in the Global South; unfortunately, that literature is sparse. The review includes research articles utilizing quantitative, qualitative, mixed methods, and case study approaches as well as literature from both research and evaluation perspectives. The review also relies on relevant literature reviews as resource documents to locate additional primary text. In consultation with a library scientist at the University of Colorado Denver, we developed a comprehensive search strategy. As illustrated in Figure 1, our preliminary search using the Auraria Libraries General Search Engine, in December 2017, produced 608 articles. Based on a review of the databases and articles identified in the initial search, we then targeted our search to the following databases: Criminology Collection, Education Collection, PAIS Index, PsycINFO, Social Science Database, Sociology Collection, and Worldwide Political Science Abstracts. This produced 426 articles. We then reviewed titles, abstracts, and key words for articles from these 426 articles. We retained articles based on the criteria that they address: evaluations of advocacy initiatives, descriptions of evaluation tools for advocacy initiatives, literature reviews relevant to advocacy evaluation, reviews/reflections on the field or history of advocacy evaluation, or articles that suggest promising outcome areas for evaluation of advocacy initiatives. This process resulted in the inclusion of 75 articles. We then conducted a hand search of the literature and additionally searched JSTOR. This process resulted in the inclusion of nine additional articles. After reading the full text of these 84 articles, the authors narrowed the literature down to 57 articles that fit the aforementioned criteria.
Advocacy evaluation literature review (1997–2017) flow diagram.
Existing Literature With Critical Reflections for the Global South
The peer-reviewed literature on advocacy evaluation in the Global South is spares, and almost none of it is written by activists or evaluators from the Global South. In our review, there were a total of six articles (two of which were published by the same author) that address human rights advocacy evaluation in the Global South and had at least one author who self-identified as being from the Global South and/or indicated at least one affiliation with an organization with operations in the Global South. This is a total of 10.5% of the identified literature; conversely, this means that just shy of 90% of the scholarly literature on advocacy evaluation is coming from, and focused on, the Global North.
Before discussing what is lacking or insufficiently developed in the existing literature, there were several themes in the literature that we feel are fairly consistent globally for human rights advocacy evaluation. These include challenges identifying short-term outcomes that are meaningful to human rights advocacy work (Egbert & Hoechstetter, 2006; Glass, 2017; Hussein et al., 2006), assessing contribution and describing attribution in reporting (Andreassen & Sano, 2007; Beer & Reed, 2009; Gardner et al., 2011; Glass, 2017; Gorvin, 2009; Hussein et al., 2006; Lohman & Amon, 2015; Montell, 2016; Morariu & Brennan, 2009; Stead et al., 2002), and the need to integrate evaluation findings into advocacy planning (Amon et al., 2015, p. 98; see also Devlin-Foltz et al., 2012; Guthrie et al., 2006; Minian et al., 2012). That said, for each one of these fairly global challenges and opportunities, the authors would suggest that distinct considerations exist in the Global South. Nonetheless, in the following sections, we emphasize a selection of these themes that we believe are imperative to consider within the context of the Global South.
In addition to these commonly reported themes, there are considerations that were either absent in the literature or that demand more or different attention in the context of the Global South. The challenges and considerations discussed in relation to the Global South stem from the postcolonialist systems which still dominate Global South realities. Further, the postcolonialist systems in which activists in the Global South operate change the context of evaluation processes significantly. These systems pervade the interactions between Global North funders and evaluators and Global South activists. At a foundational level, we propose the consideration of issues of power be treated as an essential component of high-quality evaluation, particularly when conducted in the Global South. The issue of power is embedded throughout the critical review of the existing literature presented here.
Risk and Security
The one topic that we found almost completely absent in the existing peer-reviewed literature is a discussion of risk associated with human rights advocacy evaluation. Although human rights advocacy carries risk throughout the world, the danger to activists in the Global South tends to be greater and qualitatively different. For instance, colonial rule almost always included a systematic demonization (and in many cases criminalization) of indigenous cultures. As a result, hard-won independence from colonial rule was often accompanied by a cultural renaissance in which traditional culture emerged as both a unifying motif and an instrument of social control.
Many human rights activists in the Global South are challenging fixed, idealized understandings of culture with dynamic and inclusive conceptualizations more compatible with human rights (Hudnall Stamm et al., 2004; Ibhawoh, 2000). In the postcolonial context, it has been all too easy for opponents to portray such challenges to culture as “counterrevolutionary,” insinuating that activists are working to reinstate the hated colonial oppression of the past. Such rhetoric intentionally places activists in an extremely isolated and vulnerable position, often without the meaningful legal protections taken for granted in many countries in the Global North. As opponents find legal and extrajudicial ways to close down the societal spaces in which advocacy can happen, the danger grows. This threat is significant with increasing numbers of human rights activists being subjected to lengthy incarceration, abduction, torture, assault, and murder (Amnesty International, 2019).
Unavoidably, these risks apply also to advocacy evaluators, even when they are not directly involved in advocacy themselves. Advocacy evaluation is already challenging, and in some cases, threats to evaluation participant and evaluator safety make a collection of data, data storage, or dissemination of findings virtually impossible. Security concerns impact questions of design, ethics, method, logistics, and cost. For example, Is it ethical for evaluators determined to include the voices of a persecuted minority to arrange a feedback meeting at a conference venue, knowing that (or not knowing if) venue staff often serve as informers? Should evaluators be storing data from persecuted minorities and transmitting that data electronically under conditions of threat and possible surveillance?
These and a myriad other concerns plague advocacy evaluation in insecure contexts. Often, the risks are poorly understood by evaluators, particularly those from contexts in which political and legal constraints on the abuse of power are more effective. Additionally, evaluators from the Global North working in the Global South are often protected by their relative wealth, foreign citizenship, and/or social positions of privilege while their presence draws attention and exacerbates risk to the more vulnerable people with whom they interact.
Likewise, the existing literature acknowledges that advocacy is a unique field in that it is common for activists to face active opposition, which creates obstacles to process (Glass, 2017; Gorvin, 2009; Montell, 2016). Even more relevant to risk, success can have unpredictable consequences—with any positive progress potentially inciting repercussions that change the environmental context in which the advocacy work occurs (Montell, 2016). Activists in the Global South tend to face their adversaries and these repercussions with few legal, political, or social protections. For this reason, it is even more important that advocacy evaluation in the Global South consistently considers the opposition as a factor in designing evaluation and dissemination plans.
Various methods and tools have been developed to improve the security of human rights activists (Higson-Smith et al., 2016; Rimmer, 2011; Tactical Technologies Collective & Front Line Defenders, n.d.). These tools, while no doubt invaluable to many human rights activists, have not yet fully explored the security challenges associated with human rights advocacy evaluation.
Rapidly Evolving, Unpredictable Environments
The existing literature asserts that advocacy work frequently occurs in rapidly evolving, unpredictable environments in which the intended outcomes are often equally unpredictable and elusive (Gardner et al., 2011; Glass, 2017; Jackson, 2014; Minian et al., 2012; Montell, 2016; Morariu & Brennan, 2009; Ranghelli, 2009; Stead et al., 2002). The definition of “success” often shifts with the political landscape. Many have argued that advocacy work needs to adopt more flexible evaluation methodologies to reflect how dynamic and unpredictable this type of work can be (Devlin-Foltz et al., 2012; Fagen et al., 2009; Gill & Freedman, 2014; Glass, 2017; Harris et al., 2017; Ingram et al., 2015; Minian et al., 2012).
Some of the defining features of a successful advocacy initiative should be flexibility and responsiveness (Arensman & Wessel, 2017; Egbert & Hoechstetter, 2006; Holley et al., 2014). Likewise, advocacy evaluation should value groups’ agility in being able to respond to changes in the political environment as much as their ability to stick to original strategies (Egbert & Hoechstetter, 2006). Glass (2017) explains that advocacy evaluation needs to match the dynamism of the social change process. To do this, evaluators need plans that structure in this flexibility. Accordingly, advocacy evaluators often face the challenge of developing evaluation plans that hold programming accountable, while also valuing and assessing agility (Gill & Freedman, 2014; Montell, 2016). Authors argue that evaluation plans should include built-in ways to make midcourse adjustments to the evaluation while preserving accountability to outcomes (Holley et al., 2014; Minian et al., 2012). Another way to honor the importance of agility in advocacy work is to include a measure of it; evaluators may consider developing creative ways to analyze moments when organizations pivot in response to political or other contextual changes (Egbert & Hoechstetter, 2006).
While flexible evaluation approaches are valuable for all advocacy evaluation, the sociopolitical instability in much of the Global South situates the “dynamic” advocacy environments as a central and ever-present consideration. Advocacy work in the Global South is even more unpredictable and demands highly flexible evaluations. While it is important for the evaluator or activist to know how to build in flexibility, it is essential that funders understand how to support, normalize, and account for high levels of flexibility in advocacy evaluation. It is also notable that building flexibility into an evaluation is difficult to do, and this recommendation itself assumes a certain level of technical skill and familiarity with evaluation. This further underscores the need for funders to provide support for designing flexible approaches to evaluation.
Moreover, many of the recommendations in the literature assume a level of contextual stability. For instance, scholars in the field recommend a utilization-focused evaluation approach (Gorvin, 2009). This approach emphasizes early identification of the intended uses of evaluation, and the success of the evaluation is contingent on how well the evaluation is utilized by its primary intended users. A generally utilization-focused evaluation approach can be an excellent foundation for planning an advocacy evaluation. However, several of Patton’s (2013) suggested steps for utilization-focused evaluation might be inappropriate in the context of advocacy evaluation. For one reason, several of these steps require investing a great deal of time at the front end in planning end use. This extensive preparation could ironically prove counterproductive to the agility of a campaign and the usefulness of advocacy evaluation, when campaigns will likely change course and direction fairly rapidly. In addition, steps that focus on a program model or reporting evaluation findings might not always be appropriate for advocacy evaluation. We advise integrating a utilization-focused evaluation approach that takes into account an unstable or unpredictable context and the lack of a set program model—perhaps abbreviating the initial planning and substituting a focus on continuous data use for that of “end use.”
In terms of evaluation within unpredictable contexts, the existing literature largely maintains that traditional baselines and comparisons are rarely possible and often not productive for advocacy evaluation. Most organizations lack the financial and technical resources to develop meaningful baselines against which to measure progress (Gardner et al., 2011; see also Beer & Reed, 2009). In addition, identifying (let alone measuring) relevant baselines for dynamic evaluations of highly flexible interventions in volatile contexts is problematic. Then, due to the lack of baseline data and the highly contextual nature of advocacy, comparative methodologies are unlikely to be effective (Burnell, 2008; Gardner et al., 2011; Hussein et al., 2006). Gill and Freedman (2014) assert that advocacy work is rarely able to utilize comparative, predictive evaluation approaches familiar to funders (and many evaluators). This is even more true in the Global South. With typically more unstable contexts, fewer financial resources, and more limited evaluation capacity, traditional baselines and comparisons are generally not advisable for advocacy evaluation in the Global South.
Additionally, because of how integral complex and rapidly evolving contexts are to advocacy work, the existing literature strongly emphasizes the need for evaluators to understand the historical, social, and political context in which an evaluation takes place and to use this knowledge in evaluation design. Evaluators should understand the problem being addressed by activists, consider the political environment, and use that information to inform planning, analysis, and dissemination (Cochran, 2010). Understanding political realities and the advocacy response to them sets the foundation for the evaluator to develop a realistic and useful evaluation approach.
Familiarity with the local context where an advocacy initiative is operating is critically important and often missing in a Global North system that emphasizes the expertise and “objectivity” of external evaluators (often commissioned from the Global North to evaluate Global South movements). Despite agreement that objectivity is always beyond reach, it remains an ideal of evaluators in the Global North. As a result, contextually informed knowledge is at times discredited as being “too close” to the subject. This dynamic carries the unspoken assumption that evaluators from the Global South are less capable of or less likely to adopt an appropriately critical perspective in relation to the advocacy work being evaluated. Evaluation stakeholders should expect that evaluation approaches fit the social and political context of the advocacy work and demonstrate fundamental familiarity with the local context. Indeed, local expertise should take primacy in evaluation design and implementation. Without deeply contextual evaluation design, activists are often faced with decisions to either participate in poor, even unsafe evaluations, or to face the loss of scarce resources for their human rights work.
Defining “Success”
The existing literature emphasizes that advocacy evaluators are presented with initiatives that need to define success in ways that may not look like success in program-based approaches. The value of human rights advocacy is rooted in morality, and at times, this means that initiatives choose to address situations that are understood to be largely intractable (Montell, 2016). Success in these situations may be as fundamental as having a presence and communicating solidarity, even when significant political change might be highly unlikely. This is especially salient in the Global South, where it is often more difficult to assure and protect fundamental human rights and where the related political and social situations are often declining or largely intractable (Ron et al., 2017). Moreover, human rights activists are working within these contexts with limited legal protections. It is critical for Global South activists with relevant local contextual knowledge to have the option to define no change or slowed decline as a success. This option should be available without the burden of funder expectations to demonstrate an obvious positive change resulting from their work.
Relatedly, the existing literature largely advises that human rights activists should continue to be ambitious in their efforts and to not let evaluation plans limit their drive (Coe & Schlangen, 2011; Glass, 2017; Gorvin, 2009). Evaluation should inform initiatives, not constrain them. This requires that people in positions of power are receptive to learning from evaluation findings. Accordingly, it is critical to consider the extent to which human rights activists and evaluators in the Global South believe that their views and concerns are genuinely of interest to colleagues in the Global North (McGoey & Thiel, 2018; Vogel, 2006, 2010). Although answers to this question fall across a continuum, many human rights activists and evaluators in the Global South harbor deep and enduring doubts about the likelihood of their potential contributions being heard and valued. It is not surprising that some activists in the Global South are less than enthusiastic about engaging and investing in advocacy evaluation or in sharing their efforts to learn from their work through evaluation.
Policy Implementation
Advocacy evaluators face the challenge of designing evaluations in which key outcomes may be years away. Policy change, a highly common outcome area for advocacy work, is inherently a slow process that can take years, even decades to come to fruition. Often, the most meaningful outcomes exist in the very, very long term, which poses a challenge when needing to align work with traditional evaluation plans and grant funding timelines (Minian et al., 2012; Montell, 2016; Morariu & Brennan, 2009). While this may seem a common global challenge, we argue that is actually even more problematic in the Global South.
Long-term policy outcomes are often viewed as the ultimate outcome of an advocacy campaign, but many of the most important outcomes occur after the policy change. While the time frame for advocacy evaluation poses similar obstacles around the globe, the legal assurances, protections, and safeguards for policy implementation tend to be more limited in the Global South. For instance, Klugman (2011) offers a case study on advocacy for reproductive rights during the shift from apartheid to democracy in South Africa; she shows that, while there were policy wins, organizational and resourcing challenges hindered effective implementation. This presented even longer, long-term outcomes for evaluators to consider. While policies may change, the actual protection of human rights will not occur if the policy change is enacted ineffectively. Conversely, a failed attempt to change policy might unexpectedly succeed in other ways, for example, by bringing greater public attention to an issue which might eventually result in even broader positive policy change than originally anticipated (Klugman, 2011).
Relatedly, given the complexity and extended time frame of common advocacy outcomes, advocacy evaluations should be attentive to process indicators and indicators of short-term outcomes (Amon et al., 2015; Beer & Reed, 2009; Cochran, 2010; Devlin-Foltz et al., 2012; Guthrie et al., 2006; Minian et al., 2012). Even when more obvious long-term indicators of success are elusive, initiatives may be building foundations, capacity, and solidarity that should be documented, valued, and celebrated. This documentation creates an opportunity to demonstrate progress that might be invisible with longer term indicators and more meaningfully illustrate to funders a wider range of possible successes in an advocacy initiative. It also allows activists to better understand the effectiveness of their tactics in real time, when they can use this information to redirect their efforts. Given the politically and socially insecure contexts in which activists in the Global South work, it is increasingly important to emphasize short-term goals. Within this instability, there is a greater likelihood of the need to be responsive and change course in a campaign. Setting well-planned short-term goals and conducting frequent reviews of intermediate (planned and unplanned) successes set activists in the Global South up to be able to show a wide variety of successes supporting their causes, while continuing to align the evaluation with contextual realities.
Lack of Research and Established Good Practices
Many have cited the lack of research and publishing on advocacy evaluation as a major challenge in conducting meaningful and realistic advocacy evaluation (Astramovich et al., 2017; Cochran, 2010; Gardner et al., 2011; Hussein et al., 2006; Millstein & Sallis, 2011; Minian et al., 2012; Morariu & Brennan, 2009; Rapaport et al., 2006). Because of this, there is a parallel lack of systematic evidence of what tools, methods, and approaches work best in human rights advocacy evaluation (Burnell, 2008; Cohen & Green, 2012; Gill & Freedman, 2014; Montague & Lamers-Bellio, 2012; Rapaport et al., 2006; Schmitz, 2012; Strong & Kim, 2012). As illustrated in Figure 2, human rights advocacy evaluation scholarship is emerging, although the number of published papers is still far fewer than in other evaluation fields.
Literature review scholarly article count by year.
A related point in the existing literature is that standard, predeveloped evaluation tools will almost never be what advocacy evaluators need (Glass, 2017). In the same way that each advocacy campaign is unique, each tool will need to be tailored to some extent. It is important for advocacy evaluators to become familiar with key outcome areas common to advocacy evaluation and tools used for measuring these common outcomes. We have summarized key outcome areas reported in the literature (https://www.newtactics.org/sites/default/files/resources/6%20CHOOSING.pdf, page 4). 3 It is also important that evaluators develop the capacity to adapt evaluation tools in consultation with activists and other stakeholders, including donors. Tools developed through a deeply collaborative process are more responsive to contextual and campaign factors, have greater meaning to stakeholders, and are therefore likely to generate data and results of higher quality.
In the context of a field dominated by the Global North, the need to adapt evaluation tools and approaches to each campaign is especially important in the Global South. It is also increasingly essential that local communities and activists have agency in designing or adapting evaluation to fit their context. Still, it is a technical and specific skill to be able to adapt evaluation tools for their intended purpose. Development of this skill often demands training and capacity development drawing on existing Global North evaluation practices, while prioritizing local and contextual knowledge within the Global South. This is no small feat, but it is essential to developing advocacy evaluation approaches in the Global South that support effective advocacy.
It is critical to note that there are important human rights advocacy and evaluation resources emerging about the Global South through less formalized mechanisms and in the gray literature. The gray literature includes anything publicly shared that has not received formal peer review (e.g., freely available tools, blogs, reports). The authors conducted a series of searches of the gray literature, which is difficult to do systematically (see Adams et al., 2016; Godin et al., 2015). Nonetheless, our searches suggested that gray literature on human rights advocacy evaluation in the Global South is not readily available via online searches and that which is available is limited. Firstly, the quantity of available resources was surprisingly small. And secondly, what is readily available is almost exclusively produced by evaluators and consultants in and from the Global North. For all of the aforementioned reasons, Global North perspectives on the Global South remain highly limited. Further, this indicates a need for funders to actively seek opportunities to support activists in the Global South to write and publish not only in the peer-reviewed literature but also in the gray literature, particularly on evaluation reports based on learnings from their own campaigns.
One exemplar we did find in the gray literature is titled, “A Collection of Thoughts From the Global South on Foreign Policy and Human Rights: Experiences and Strategies From the Field” (Conectas Direitos Humanos, 2018). This piece was produced by Conectas Direitos Humanos based in Sao Paulo, Brazil. It outlines a series of case studies of advocacy initiatives from Brazil, Mexico, Nigeria, South Africa, Mongolia, Indonesia, India, and Maldives. While this article is not specifically on the topic of advocacy evaluation, it does offer recommendations for indicators to track progress on human rights advocacy. These types of resources and tools coming from the Global South are better equipped than those coming from the Global North to take into account the many critical considerations about advocacy evaluation in the Global South.
Notably, this exemplar paper in the gray literature argues that scholarly work has a critical role in influencing policy makers and citizens, which is a fundamental argument of this article: Perspectives from the Global South need to be included in the scholarly literature in order to develop a relevant and mature global field of advocacy evaluation. While it is important for the experiences and expertise from the Global South to emerge through the gray literature, there is also good reason to want this work to be included in the peer-reviewed canon. First, the gray literature is not systematically organized, and, as such, these types of resources are often difficult to locate and depend on informal networks for dissemination. Second, funders tend to take peer-reviewed literature and expert Northern evaluator perspectives into account when deciding on approaches to evaluation. While we believe this should and could be different, it is not yet the case. Lastly, even with its serious limitations (e.g., Northern bias, English-language bias, academic bias, accessibility inequity), scholarly literature does have a formal review process that seeks to assure the integrity and quality of the publication. Conversely, gray literature has no such standardized process, and it is up to the reader to sift through information of highly varying quality, often without the resources, time, or technical capacity to do this well. We maintain that it is essential both to highlight and disseminate gray literature and for the scholarly literature to catch up to and integrate evaluation learnings and innovation happening in the Global South.
Poor Resourcing and Assumed Capacity
Underlying all of these challenges is the fact that advocacy initiatives are frequently underresourced and have limited internal capacity to support evaluation work (Glass, 2017; Harris et al., 2017; Jackson, 2014; Minian et al., 2012; Montague & Lamers-Bellio, 2012; Montell, 2016; Ogilvie et al., 2008). Therefore, finding able, willing, and committed evaluation champions within an advocacy organization or community can be trying (Ogilvie et al., 2008; Ziabakhsh, 2015). Additionally, because of the innovative approaches required of advocacy evaluation, organizations may feel ill-equipped to conduct these challenging evaluations. Again, this is especially true in the Global South. Resources, capacity, and therefore readiness for the complexities of human rights advocacy evaluation are often even more limited. There is a need for greater funding, collaborative learning, and highly usable, adaptable evaluation tools designed specifically for work in the Global South.
The literature is replete with promising approaches that assume high levels of evaluation capacity, financial support, and available time, all resources which are not often available in the Global South. This literature is not written for activists in the Global South; it is written for trained evaluators in the Global North. Indeed, the authoritative text in the field asserts that the content of the book is “easy to use” while bookending with a disclaimer a few pages later that “a basic understanding of evaluation is assumed” (Gardner & Brindis, 2017, pp. vii, xi). In the review of literature, for instance, scholars frequently suggest mixed-methods approaches to advocacy evaluation (Harris et al., 2017). While certainly a helpful approach to address some of the constraints within advocacy evaluation, it is not useful advice to an activist without technical evaluation or methods training. Perhaps even more of an ill fit, the most commonly described promising approach for advocacy evaluation in the existing literature is developmental evaluation. This approach emphasizes flexibility, systems thinking, rapid response, and adaptability, and relies on time-intensive implementation by a highly skilled/experienced evaluator (see Patton, 2010). While these are appropriate and helpful resources and recommendations for an assumed audience of trained evaluators in the Global North, they are often inappropriate for the activist in the Global South looking for relevant and usable guidance on evaluation.
Certainly, we need this perspective and these recommendations in the literature, but we also need literature providing feasible guidance for those initiatives that do not have those same resources. When writing to the vast majority of individuals interested in conducting human rights advocacy evaluation in the Global South, one is speaking predominantly to nonevaluation specialists. They are often experts in their main line of work but are tasked to also conduct advocacy evaluation, reporting to donors, or strategic learning for their campaign. Guidance on advocacy evaluation for these activists needs to be accessible, concise, and highly usable for a nonevaluation specialist. If that seems a tall order for the evaluators reading this article, imagine how it feels to be an activist in the Global South tasked to conduct an evaluation, premised on Global North assumptions, and provided highly technical resources to accomplish it. Add to this situation another complication if you are an activist operating without English as a first language. Our suggestion is that until such time as human rights activists in the Global South have access to local evaluation resources similar to those shared by their colleagues in the Global North, the burden of managing expectations of advocacy evaluation should lie with stakeholders in the Global North. It is for the highly resourced actors of the Global North to find ways to support activists in the Global South to design and conduct contextually appropriate evaluation that advances their advocacy work, while imposing as little as possible control in the form of evaluation requirements.
Donor Communication
Existing literature describes a tension between a “desire for understanding…the impact [of an advocacy intervention] and the need to learn quickly to support strategic decisions” (Beer & Reed, 2009, p. 153 4 ). This tension is often acutely felt between funders, evaluators, and activists. Because of this, it is increasingly important to intentionally and transparently develop a shared understanding of the evaluation objectives (Devlin-Foltz et al., 2012). The literature identifies two key areas for ongoing communication. First, communicating the value of flexibility and adaptation is necessary for creating a funding environment that supports meaningful and effective advocacy evaluation (Egbert & Hoechstetter, 2006). Second, it is critical to provide donor education about the importance of process indicators, as well as mid-term and short-term outcomes, in addition to long-term outcomes such as policy change (Egbert & Hoechstetter, 2006). Ongoing, transparent dialogue between funders and evaluators, on these and other aspects of the evaluation, is essential for effective advocacy evaluation.
With the inherent power differential between funders, evaluators and activists, we know this type of open dialogue is fraught and rarely actualized. Then, one must think of the several additional power differentials inherited from multiple global systems of oppression dividing Global North funders from Global South activists. Layered on that, there is another power differential between any external evaluator from the Global North and a funded initiative in the Global South. To begin, there are the historical, systems-level inequalities built into any North–South relationship. Then, there is the tremendous differential in resourcing; many activists in the Global South work with little to no funding, and, as such, an interested funder is incredibly powerful. The power of a Global North funder leads to enforcement of Global North–centric expectations. For instance, grantees may be required to speak and write in English. This excludes many groups and organizations. This also forces the potential and actual grantees to engage in critical and highly technical components of this funder relationship in a second, third, et cetra language, which may unfairly shape how the funder views the work itself and, in some cases, actually change the direction of the work. In addition, most tightly linked to this present article, funders are largely advised by evaluators from the Global North, who are rarely adequately versed in the local communities on which they are advising on. Moreover, there are few tools that specifically address the needs of those working in the Global South. It is entirely unfair and unrealistic to place the burden of bridging this power chasm on activists in the Global South. Instead, it should be the obligation of funders and evaluators in the Global North to do so. High quality and well-utilized evaluation is a powerful form of knowledge that can be used transformationally by human rights activists. However, Global North systems are often designed to use evaluation to preserve the current power structure, instead of allowing an evaluation’s use to challenge established ways of knowing and being.
Advocacy Evaluation in the Global South
While the few cited authors from the Global South have provided important contributions to the literature, the near absence of evaluators and activists from the Global South is a critical challenge for the global advancement of advocacy evaluation as a field. When coding the overall literature on advocacy evaluation, we looked for consistent themes across the full range of articles included in the review. Conversely, since there are so few articles from the Global South, instead of looking for themes, we looked for ideas that were unique to these settings or more prominent in these publications than in the literature from the Global North. In this section, we review the themes that were emphasized in this subset of literature.
First, the literature from the Global South on advocacy evaluation places more emphasis on the deleterious effects evaluation can have on advocacy work. For instance, Gorvin (2009), writing about Human Rights Watch, warns us not to allow evaluation outcomes to limit advocacy initiatives. They explain, “there is an element of moral imperative behind parts of our work—we cannot simply commit to do just what is ‘profitable’ in terms of impact” (Gorvin, 2009, p. 483; see also Montell, 2016). These authors highlight the risk of evaluation prioritizing what is achievable, measurable, and appealing to funders over what is the right and important work to do. Underlying this is a disconnect between activists on the ground in the Global South and influential actors, especially funders, in the Global North setting expectations and giving guidance that is founded on very different contextual understandings.
Coates and David (2002), who make distinctions between nongovernmental organizations in the Global South versus North, warn that evaluation findings can simply provide bad information that undermines advocacy work; “Standardised forms of [evaluation] are likely to be inappropriate—they will probably provide misleading information, and may create perverse incentives that undermine joint action” (p. 530). Moreover, they identify advocacy evaluation as a market that is being pursued for profit without a real sense of how it should be done. The tone of the text is illustrative of concerns about the field of evaluation in advocacy work. For instance, The “market” (of internal staff, research institutes, and/or consultants) is just starting to respond, recognising that M&E is set to become a growth industry. The multinational consultancy agencies are pursuing the potentially lucrative sector of monitoring the impact of companies on workers, local communities, and the environment, and attempting to establish themselves as credible verifiers. Smaller consultancy firms that have experience in M&E for project work are now eyeing the potential for evaluating advocacy activities. Meanwhile, NGOs are adding internal staff and starting internal training and capacity building in M&E for advocacy. Yet a second problem remains: how do you do it? (Coates & David, 2002, p. 531)
An inevitable result of the lack of trained evaluators in the Global South is that advocacy groups in these parts of the world often do not employ trained evaluation staff. Instead, project managers or administrative staff are tasked with monitoring and evaluation functions. These staff often lack the time and skills to design evaluation frameworks that do justice to complex advocacy projects. Until funding organizations are prepared to support evaluation training and provide mechanisms to properly resource evaluation functions, thereby initiating a grounded advocacy evaluation movement in the Global South, the practice of advocacy evaluation is unlikely to flourish in these parts of the world, let alone have additional capacity to submit manuscripts for peer review.
The field of advocacy evaluation is relatively young, and tertiary education institutions are often slow to integrate new approaches, particularly those perceived as critical of more traditional, accepted methodologies. The ongoing academic imperative to decolonize social science training in the Global South (Bhatia & Priya, 2018; Ratele et al., 2018) can be helpful in this regard. This work encourages Southern training institutions to “leapfrog” over the conservative delay in integrating innovative methodologies by emphasizing the importance of advocacy evaluation approaches that prioritize the interests of vulnerable people and are responsive to the contextual realities of Global South communities. While such endeavors may be supported by Northern partners, the work of educational decolonization can, by definition, only be advanced by Global South scholars.
Third, this literature on the Global South places particular emphasis on the ethical use of advocacy evaluation data. In an article on human rights advocacy on sexual violence in Liberia, Cohen and Green (2012) emphasize the misuse of data and the potential consequences. They report having observed that in some instances, “[a]dvocates do not simply report findings, but often reinterpret and dramatize facts in order to produce public outrage” (Cohen & Green, 2012, p. 446). While it can be argued that it is part of the role of both activists and evaluators to present information in a way that does raise public awareness (and indeed outrage) about problems that are otherwise minimized or invisible, this should not happen at the expense of the accurate and valid reporting of empirical evidence. Apart from the ethical problem with the misuse of evaluation data, these authors point out that such action might in fact be self-defeating. Over time, we hypothesize, information strategies based on suspect facts may lower advocates’ credibility and contribute to a sense of intractable, inexplicable crisis…. [And] may decrease the attention and resources directed to future [social problems]. (Cohen & Green, 2012, p. 446)
Fourth, the authors also emphasize that success needs to be defined differently for human rights advocacy work, with a strong emphasis on including “lack of deterioration” or even “slowing deterioration” as possible dimensions of success. As one author explains, “[a]ttributing positive change to our own work is often uncertain; even more so is attributing the absence of negative change (i.e., things did not get worse)” (Gorvin, 2009, p. 478). Another asserts that “[i]t is even more difficult to measure our role in preventing deterioration of human rights, although this is also an important role of human rights advocacy” (Montell, 2016, p. 930). That author goes on to explain that, While I do not have a way to evaluate whether the situation would be even worse without the work of human rights organizations, it is likely their documentation, litigation, and reporting during times of crisis act as a deterrent to even harsher human rights violations. (Montell, 2016, p. 963)
Conclusion
In this concluding section, we provide recommendations to stakeholders in the global human rights advocacy evaluation community to address the pressing challenges for this work in the Global South. We focus on the development of tools and resources; attention to ethical, methodological, and logistical challenges in the Global South; and decolonizing systems of evaluation and capacity building. These recommendations can be accomplished through coordinated efforts of Global South activists, evaluators, funders, and educators, and supported by efforts of their Global North peers. The authors outline more specific actions that can be taken by each of these communities to address these recommendations.
For Global South Evaluation Practitioners
There is a lack of research on human rights advocacy evaluation coming from activists in the Global South in large part due to tremendous global power inequities. These inequities continue to perpetuate this weakness in our field. For that reason, we place emphasis on what those who hold power and who perpetuate systems of inequality (funders and evaluators in the Global North) must do to advance our field in subsequent sections. That said, we offer two specific reflections for activists conducting evaluation in the Global South.
First, we encourage activists to share what they learn however they are able; the field of advocacy evaluation and other colleagues in the Global South need these insights. There is a tremendous need for more published work on advocacy evaluation in the Global South, written by the evaluators and activists doing that work. It is critical that practitioners in the Global South share what they learn. There are many activists and advocacy evaluators who are eager to learn from others doing this work. Dissemination of this work can provide a solid foundation of advocacy evaluation that serves advocacy in the Global South. In addition to peer-reviewed publications, it is important to consider additional forums for sharing work and expertise, such as conferences, donor meetings, blogs, periodicals, and social media, among others.
Second, we also encourage activists to familiarize themselves with the risks introduced by evaluation to themselves, their campaigns, and the communities they are supporting. This is a fundamental way activists can seek to uphold the human rights principle of doing no harm when tasked to conduct the evaluation. We encourage activists in the Global South to carefully consider risk and security associated with human rights advocacy evaluation. This advice is largely absent in the literature. While activists are typically acutely aware of security concerns associated with human rights advocacy, the risks associated with evaluation are almost always less familiar. We strongly encourage activists to consider the risks associated with data collection, storage, and reporting. Consider how data collection might put activists, evaluators, and communities at risk through exposure in the community, to officials, or even within the household. Also consider the security of the medium by which that information is collected (i.e., in person, telephone, online, paper). Additionally, activists should consider what the risks might be if someone were to access the data they are collecting and storing. Campaigns often have active opposition who may be highly motivated to access evaluation data. Lastly, we encourage activists to consider the ethics of reporting on evaluation with particular attention to protecting the people being evaluated and on the accuracy of the information. These reporting considerations are important both for the individual’s security and for the success of the campaign itself. We recommend reviewing this advocacy evaluation risk assessment document when planning evaluations ( https://www.newtactics.org/resource/advocacy-evaluation-and-principle-do-no-harm ). Activists also have an opportunity to advocate funders and other stakeholders about those ethical boundaries and the protections necessary for the activists themselves, the communities, and the campaign. When possible, remind funders of human rights work that they should themselves encourage and support this type of risk assessment as well as help address the identified risks introduced by their evaluation requirements.
For Funders of Global South Advocacy Evaluation
The advancement of effective Global South advocacy evaluation requires that Global South activists and local advocacy evaluators are funded to advance that work. Funding proposal guidelines, expectations, norms, and evaluation criteria profoundly shape the landscape of evaluation. There is not currently a consensus in the field about good practice in advocacy evaluation and certainly not in the context of the Global South. As such, funders are put in the difficult situation of setting expectations for advocacy evaluations without guidance from the field. Based on the current literature and our own experiences working in the Global South, we offer some recommendations to funders.
We encourage funders to consider ways in which they can learn from and give a voice to activists and evaluators in the Global South. One powerful way to do this is to support meetings for advocacy evaluators and activists working in the Global South in order to convene and learn from one another. This is an essential mechanism by which innovation can be fostered and transmitted. It is important to prioritize sharing and learning among Global South practitioners; these insights will have the most direct relevance. Insist that conversations about work in the Global South are directed by significant representation from the Global South.
Funders should also develop requests for proposals and requirements for evaluation plans with the lived realities of advocacy work in the Global South in mind. First, encourage responsiveness to external change. Adapting to changes in volatile political and social environments is a critical indicator of success, much more so than adherence to preplanned activities. Second, encourage the use of formative evaluations with clear connections to programmatic learning and rapid adaptations based on evaluation findings. Strong advocacy evaluation plans will include structured opportunities for making midcourse adjustments to the evaluation. Third, prioritize simple and innovative evaluation plans over complex and poorly fit, more familiar approaches. The oft-proposed comparative, predictive types of evaluation plans are rarely feasible or appropriate for advocacy evaluation and can constrain the effectiveness of an advocacy initiative.
Fourth, when a campaign’s context involves a trajectory of deterioration, lack of change should be an acceptable outcome. Fifth, require that risk and security concerns are included within evaluation plans and implementation. Evaluators should be required to demonstrate their familiarity with the local context and the extent to which they can assess risks posed through data collection. Evaluations that do not address risk should be required to do so as a prerequisite for funding. Sixth, consider supporting the development of internal evaluation capacity within Global South campaigns and organizations over contracting external evaluators from the Global North. Alternatively, support activists to identify and contract with a local external evaluator. Furthermore, since advocacy evaluation is an emerging field, it is advisable to generously support capacity building in grant proposals and project budgets. Lastly, funders could remove the burden on Global South practitioners of needing to be able to write in English by allowing more linguistic diversity in projects they fund and the required deliverables. The burden of translation should be instead shifted to the better-resourced funder or, conversely, translation fees for grantee–donor communication should be included within funding structures.
For Evaluators and Academics in the Global North
Given existing global dynamics, evaluators and academics in the Global North have an advantage and may find themselves in a position to benefit personally and professionally from the current need for strong advocacy evaluation and limited evaluation capacity in the Global South. It is worth noting, however, that they are also in a position to support the emergence of a grounded movement of Global South advocacy evaluation, which will enrich this field of evaluation exponentially, accelerate advocacy movements, and more equitably distribute power and resources. The following are ways in which the Global North evaluation community can support this emergence.
To begin, we encourage evaluators and academics in the Global North to genuinely partner on funding opportunities with Global South individuals and organizations. Within these partnerships, we encourage the partners in the Global North to seek opportunities to give partners in the Global South more authority, power, and voice in the relationship and in the work. Second, allow current beliefs about evaluation to evolve based on what is learned from Global South advocacy evaluation practitioners. The context and values underlying work in the Global North should not define what is determined as high-quality evaluation. Third, insist that Global South colleagues are meaningfully included in relevant conferences and meetings. When discussing advocacy evaluation in the Global South, the conversation should be undertaken primarily by advocacy evaluation practitioners within the Global South. This could mean intentionally diminishing our own platform, status, and/or influence in these conversations. Fourth, when conducting advocacy evaluation in Global South contexts, be aware of the risks inherent in data collection and take responsibility by insisting on protections for participants and advocating for the resources to conduct high-quality, low-risk evaluations.
As a community supporting and conducting human rights advocacy evaluation in the Global South, we all have a role to play in developing evaluation approaches and resources that are relevant, usable, and safe. Funders can create the space for on-the-ground learning and sharing back to the field. Evaluators in the Global North can forge true partnerships that foreground the innovative work happening in the Global South. In addition, activists and evaluators in the Global South can share their experience and expertise to move the field forward and forge more relevant, usable, and safe evaluation approaches. A critical challenge for the advancement of advocacy evaluation is to bridge the broader field of evaluation, the field of advocacy evaluation, and the lived realities of advocacy in the Global South.
Footnotes
Acknowledgments
Thank-you to the staff of the Center for Victims of Torture (CVT)—specifically the New Tactics in Human Rights program and CVT’s Research Department—for contributing their experience and expertise to the development of this article. We would like to recognize The Evaluation Center at The University of Colorado Denver for their critical contributions to review of the literature on advocacy evaluation.
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: These materials were made possible through a grant from the U.S. Department of State Bureau of Democracy, Human Rights, and Labor.
