Abstract
This paper unpacks our efforts as external evaluators to work toward decolonizing our evaluation practice. Undertaking this writing exercise as a form of reflective practice demonstrated that decolonization is much more complex than simply translating materials, organizing locals to collect data, and building participants’ capacity around Western modalities. While this complexity is clear in the decolonization literature, practice-based examples that depict barriers and thought processes are rarely presented. Through this paper, we deconstruct our deeply held beliefs around what constitutes good evaluation to assess the effectiveness of our decolonizing approach. Through sharing our critical consciousness-raising dialoguing, this paper reports our progress thus far and provides information and provocations to support others attempting to decolonize their practice.
Introduction
Decolonization is a movement to liberate people from imperial colonization. It is “a way of thinking, knowing, and doing” that surfaces our positionality and seeks to restore equilibrium to structural power imbalances resulting from colonization (Ndlovu-Gatsheni, 2015, p. 485). This is a deeply complex process that necessitates lifelong introspection and authentic commitment toward power redistribution and understanding. Decolonization aims to unsettle and dismantle the white Western structure that underpins global dynamics and systems; as such, it is much more than a project to ensure cultural appropriateness (Cavino, 2013), bottom-up programming, or social justice (Tuck & Yang, 2012).
Some decolonial thinkers argue that reform tinkering at the edges of colonization is ineffectual and that vast structural transformation is required for authentic change. While we agree that a structural focus is necessary, our limited ability to impact the macro level led us to focus on what we can change in our roles as external evaluators—ourselves and our practice. This harks back to Ngũgĩ wa Thiong’o's (1986) call to decolonize our minds.
While decolonization theory is strong, its adoption at the practice level in the external evaluation of international development programs is immature, and decolonization as a concept is being (mis)used in reductionist ways that dilute its meaning (Tuck & Yang, 2012). This is not due to a lack of appropriate evaluation methodologies, as many alternative evaluation approaches have been shared and developed in response to the largely colonial role of evaluation, dissatisfaction with donor-driven approaches, and recognition of the central importance of community-led processes (e.g., Chilisa, Major, Gaotlhobogwe, & Mokgolodi, 2016; Fetterman & Wandersman, 2005; LaFrance & Nichols, 2009, 2010). Despite the existence of alternatives, standard approaches prevail in practice and written records of practice-based examples offering reflective consideration of decolonization are limited.
This story began with the first author/external evaluator (LK) being approached by a small, Australian-based non-government organization (NGO), which runs a community development program in Rakhine State, Myanmar through a local Rakhine office staffed with a Rakhine workforce. The brief was to conduct four evaluations over the three-year funding cycle, in line with the European-based donor's requirements. The inherent power imbalances at the outset are vast and raise a multitude of questions regarding the development sectors’ perpetuation of colonialism, which are critiqued at length elsewhere (e.g., Durokifa & Ijeoma, 2018; Escobar, 1995; Kothari, 2006; Langdon, 2013). However, the broader programmatic and funding aspects of the NGO's operations are beyond the scope of this paper, which focuses solely on the external evaluation. Despite being out of scope, and despite dedicated organizational commitment toward locally empowering, co-designed, and community-led programming, the colonizing impact of the NGO and donor, as well as the colonizing impact of the development sector more generally, is inextricable from the evaluation of the program. LK was reluctant to take the role, feeling it would be better filled by a Myanmar national. However, with the baseline report deadline looming, the contract was agreed as long as hiring a national was prioritized with the aim of LK moving out of the role before the third report. In response, a Myanmar national (PPTH—the second author/external evaluator on this paper) was secured within the first year of the contract with a one-year handover plan.
To briefly outline our positionality upfront, PPTH has Rakhine heritage but was born and raised in a rural area of a neighboring state in Myanmar. She speaks her local language and Burmese but does not speak Rakhine. She is temporarily located in Australia while undertaking her doctorate and then intends to return home to Myanmar, political situation pending. PPTH was new to evaluation when commencing this project. LK is an English-speaking Caucausian who is embedded within the Western education system. She has 18 years experience as an evaluator and seven years experience working on research and evaluation projects in Rakhine State.
Couched in international and community development, this paper explores assumptions and tensions around power and control, specifically focused on the external evaluation component of the relationship between the Australian-based office and the Myanmar-based program site. By telling the story of external evaluation as a case study, this paper provides ideas for rethinking evaluative processes in light of power imbalances and for using evaluation as a mechanism for decolonization. This approach to external evaluation seeks to enhance the value of evaluation for local people and restore control over evaluative processes and data, and the learning inherent in those, away from external actors. This approach aims to remove power from people foreign to the country where the community development program is implemented and invest in recognizing, prioritizing, and supporting the capacity of local actors to undertake those roles.
This practice-based paper offered us (as external evaluators) an opportunity to rework and deconstruct our deeply held beliefs around how evaluation should be done to answer the question of how effectively we implemented a decolonizing approach to evaluation, using our case study as a worked example. As such, this paper reports our progress thus far. It provides information to support other evaluators attempting to work toward decolonizing their practice through sharing our critical consciousness-raising dialogue. After reviewing the decolonization literature, describing the context and program under evaluation, and explaining our approach toward decolonizing evaluation, we critically analyze our progress. Through this exercise we came to realize that decolonization is a much deeper and more complex process than simply organizing locals to facilitate data collection, seeking their input, translating findings, and building capacity around Western ways; it is a never-ending, highly complex personal and professional process that requires ongoing commitment and introspective reflexivity (Ngũgĩ, 1986). We close with implications and considerations that could help other evaluators seeking to go on the journey of decolonizing their practice.
Literature Review
There is a well-established canon of literature on the aims and effects of colonialism. At its most simple, colonization is understood as the economic expansion and takeover of the physical assets of others. At a deeper level, this includes subjugation and the imposition of Western ideas and control over all aspects of others’ lives (Childs & Williams, 1997; Fanon, 2002/1961). These aspects go beyond tangible assets of lands and liberty to include the subjugation of culture, language, and knowledge (Tuhiwai Smith, 2021). Colonization disconnects people from their social hierarchies and structures, means of communication, histories, land, and ways of being, knowing, and doing. As a result, the lives and ecosystems of colonized people are thrown into complete disorder as their ways of functioning are systematically destroyed and deconstructed (Ahluwalia, 2001; Nandy, 1989).
Through the means of colonization, and the so-called scientific discoveries, economic exploitation, and framing of certain peoples as superior or inferior, the West has successfully implemented and perpetuated ideological claims that white skin is best and Western civilization is superior (Elias et al., 2021; Fanon, 2002/1961). Rather than ending with the dissolution of the British Empire post-World War II, colonization continues due to the deep, internalized, systemic effect of Western dominion over other peoples, which has global intergenerational reverberations for the ways humans perceive themselves and others as well as how they understand their place in the world. As colonization has fundamentally transformed global society, the actions of individuals to address it can be viewed as tokenistic. However, as individuals, we do not have the power to dismantle this pervasive structure, but we can make a small contribution by seeking to decolonize our minds and using our thoughts and questions to bring others on the journey. Within this scope, we focus on unpacking how the pursuit of evaluative knowledge is deeply embedded in complex layers of imperial power and colonial practices.
In a colonial context, research has been used to usurp knowledge and justify colonial practices (Tuhiwai Smith, 2021). Therefore, research and evaluation are undeniably also about power and domination (Chilisa, 2019). In addition to this, research and evaluation are institutionalized and interwoven into the fabric of power structures whereby evidence, produced via certain methodologies, conducted through certain institutions, by suitably qualified personnel, is used to promote understandings that influence operations, including across the development sector (Tuhiwai Smith, 2021). Decolonization of evaluation, then, should be a process that intersects with colonialism at multiple points and questions the structures that perpetuate colonizing practices, cultures, and norms. This highlights the enormity of the decolonization project and struggle, which influences and pervades every aspect of humanity.
Many scholars have analyzed the colonizing effects of research (e.g., Estrada, 2005; Grande, 2008; Tuhiwai Smith, 2021). While scholars have presented alternative frameworks that explicitly seek to decolonize research, others have shared additional approaches and methodologies for evaluation (e.g., Brown & Di Lallo, 2020; Cavino, 2013; Chilisa, 2019; Kawakami et al., 2007; LaFrance & Nichols, 2009, 2010; Waapalaneexkweew, 2018). In evaluation, colonization manifests as judgments of merit, value, worth, and significance that are often defined from perspectives that are culturally and geographically distant from the perspectives of the evaluation subjects. These determinations are regularly applied by outsiders without consideration of local values and culture, let alone inclusion of locally devised and led processes. Evaluation is colonized when outsiders design the program, define outcomes and indicators of success, pose evaluation questions, gather and analyze data, and formulate results in ways that ignore and silence Indigenous and local voices. The colonized nature of evaluation largely continues in practice without sufficient critique and challenge of power structures and domination (Hopson et al., 2012). The decolonization of evaluation requires the decolonization of knowledge, culture, power, and politics as well as decolonization of the evaluation methodology itself, as introduced in the following sections of this literature review. Consciousness-raising around these topics can lead to better praxis (Ngũgĩ, 1986).
Decolonizing Knowledge and Culture
As Tuhiwai Smith (2021) argues, knowledge belonging to and about colonized people is exploited and extracted by colonizers; as such, knowledge is mined like a commodity and stolen by colonizers in the same manner as minerals and other resources. Colonizers interpret this knowledge and value it according to Western frames. Through the authority of their representations, they put forth a certain perspective of colonized people that justifies the colonization, oppression, and deconstruction of existing local systems. These understandings infiltrate the cultural archives of colonizer and colonized to remake reality and reframe colonized people as passive objects of study rather than as knowledge holders and creators in their own right. As a result, colonizers’ discovered knowledge is stored, organized, classified, theorized, and constituted as research and evaluation.
Colonization has been imposed through different mechanisms, such as education (Shizha & Kariwo, 2011). Through education as a colonizing process, non-Western people are fed uncritical information dominated by Western knowledge and ideology (Chilisa et al., 2016). This uncritical knowledge includes the infiltration of Western ways of understanding research and evaluation, and Western ways of categorizing and valuing different forms of evidence. In this, understandings of research and evaluation are diluted and diminished, focused exclusively on Western understandings that limit our ability to conduct these activities with the richness of diversity.
To decolonize evaluation, local knowledge specificities, preferences, and practices are required. Further, decolonizing evaluation means that local values and knowledge are used to develop standards of what constitutes quality evaluation and trustworthy evidence, although some counter that “the rigour of Western […] scientific validity of research and evaluation should never be compromised” (Cloete & Auriacombe, 2019, p. 8). While recognizing local ways and rethinking of dominant norms is largely positive, and will help develop colonizers as well as empower the colonized, it is important that in our enthusiasm, the West does not use this as another opportunity to usurp Indigenous/local belongings and remake them as our own.
Decolonizing Politics and Power
The production of knowledge is highly political; as such, evaluation is politically imbued in cultural and power structures (Chilisa et al., 2016; Eyben et al., 2015; Kelly, 2021a). As Foucault (1977) outlined, our ontologies and epistemologies are couched in power, politics, and practices that constantly shift and transform across time and place. Colonization played a heavy role in the transformation of ontologies and epistemologies in colonized communities whereby the West imposed its power as the maker of legitimate knowledge. As such, colonized people have been exposed to globalized ways of being, knowing, and doing that are grounded in the hegemony of the West (Tuhiwai Smith, 2021; Wilson, 2008).
Recognition of the politics that underpin evaluation highlights how evaluative knowledge reproduces and perpetuates power dynamics between a local context and the dominant Western culture (Chilisa et al., 2016; Eyben et al., 2015). This is particularly pronounced in the evaluation of international development programs and projects, as development itself is heavily intertwined with colonizing practice (Escobar, 1995; Langdon, 2013). Thus, the decolonization of evaluation requires analysis and reflection led through the perspectives of local culture and values in a manner that restructures power relations around the construction of evaluative knowledge. Further, regardless of evaluators’ positionalities, it is important that they consider the politics of knowledge construction and reflect on what, how, and why they privilege certain forms of knowledge, noting the potential internalization of colonizing ideologies (Kovach, 2010).
Decolonizing Evaluation Frameworks and Practices
The evaluation discipline is dominated by Western theory and practice (Cloete & Auriacombe, 2019). Evaluators seeking to decolonize their practice, us included, have focused on translating evaluation tools and indigenizing data collection methods without addressing the deeply held beliefs that underpin the theory and practice on which we base our evaluative knowledge. We outline some examples of these colonizing and decolonizing beliefs in Table 1.
Examples of Colonizing and Decolonizing Beliefs.
A Context of Violence and Poverty
The external evaluation utilized as a worked example in this paper was commissioned by an NGO comprised of a head office in Australia and a country office in Myanmar. The focus of this paper is on the organization's flagship program, which operates in Rakhine State, Myanmar and is overviewed in the subsequent section. Rakhine State has a history of colonization, with annexation by Burma before and after British rule (1826–1948) and continues to experience intractable conflict and extreme poverty (Ware & Laoutides, 2018).
While conflict has simmered for centuries, it erupted into intercommunal violence in 2012. At this point, the conflict was largely constrained to fighting between the Muslim Rohingya and Buddhist Rakhine and resulted in 140,000 people being displaced across the state (UNOCHA, 2013). The conflict intensified between 2015 and 2017 when the Myanmar military became involved in meting out disproportionate repercussions for small-scale attacks committed by armed Rohingya and Rakhine insurgent groups. This resulted in a bloody tripartite conflict where conflict between the Rohingya and Rakhine was exacerbated for both groups by additional violent conflict with the military. The military coup in February 2021 added a further level of insecurity. The complexity of this situation is beyond the scope of this paper and is well covered elsewhere (e.g., Ware & Laoutides, 2018). However, it is important to note that the conflict is unresolved, continuing, and has resulted in the death and displacement of thousands of Rohingya and Rakhine people. Additionally, while the Rohingya people are also from Rakhine, throughout this paper the moniker “Rakhine” refers to the Arakan/Rakhine ethnic group who are largely Buddhist.
The Rohingya population in Rakhine State has been decimated by the conflict with less than 30% of their pre-2015 population living in the state outside of internal displaced persons camps and Bangladeshi refugee camps (Ware et al., 2022). The small number of Rohingya remaining in villages, estimated at less than half a million, subsist in a dire situation of severe poverty, oppression, and insecurity.
While over two million ethnic Rakhine reside in the state, and are relatively better off than the Rohingya, they too face insecurity and discrimination from the Burmese authorities and live in fear of their Rohingya neighbors and the Myanmar military. Although both the Rakhine and Rohingya people suffer poverty, oppression, and insecurity, these are greatly amplified for the Rohingya who are confined to the areas surrounding their village and have their freedom of movement curtailed or limited, are not entitled to citizenship and the rights and services that accompany citizenship, and are generally disenfranchised, hated, and persecuted.
Poor communication and infrastructure, low education levels and high illiteracy, language barriers, poor relationships between ethnic groups, patriarchy, insecurity, federal prohibitions, high disaster risk, armed intra-communal conflict, pandemic, and the military coup all contribute to the complexity of conducting programmatic work and, by extension, evaluation in this region. The community development program (evaluand) is briefly outlined in the following section after which we reflect on moving toward a decolonizing approach to evaluation in this challenging setting.
The Community Development Program
While largely out of scope and tangential to this paper, a brief overview of the evaluand is presented here to contextualize the evaluation. The community development program supports communities to identify, advocate for, and act toward fulfilling their own needs and is committed toward co-designed, community-led programming. The goal is that communities are empowered to drive their own development, advocate to local government for investment and services, and cultivate peaceful relationships to heal social cleavages. As well as a focus on development, the program targets strengthening local capacities for peace between ethnic Buddhist Rakhine and Muslim Rohingya communities. For this reason, villages were invited for inclusion in the program according to their position as a pair of proximate Rakhine and Rohingya villages. The current cohort of villages taking part in the program consists of 12 village pairs (24 villages).
The program acts as a community organizer by working with communities to help them identify a village pair representative (the facilitator) who then attends quarterly community development training in the state capital of Sittwe, brings that knowledge back to share with their village pair, and helps organize community-led projects. This training covers subjects such as human rights, inclusion, diversity, identity, and everyday peace and is delivered in a highly participatory manner utilizing arts-based methodologies. A key aspect of the training is working with trainees to reflect on their identity and anti-Rohingya attitudes and assumptions. As current government restrictions prohibit the movements of the Rohingya, all facilitators attending training in Sittwe are ethnic Rakhine. In a first step to ameliorate this challenge, the program has hired additional local trainers who travel to provide direct training to Rohingya villagers.
Facilitators are paid a salary by the NGO and some funds are available from the NGO for village projects. However, villagers are encouraged to seek funding from within their village and their local government whenever possible and utilize NGO funds as a top-up source. Many of the existing and completed community-led projects have been funded by village residents and have included in-kind contributions with the residents working together on mutually beneficial initiatives including infrastructure, education, and community event projects.
Evaluation Approach
External evaluation of the case study program had been conducted by a foreign external evaluator in its previous iterations over the past decade. When the original evaluator moved into a different role in early 2019, the NGO sought a replacement and looked for an external evaluator with knowledge of the context in Rakhine State. The focus on finding a highly qualified and experienced replacement evaluator was probably influenced by subtle, potentially subconscious assumptions around evaluation quality and rigor as well as around commitments to donors. Previous research has demonstrated that NGO staff regularly assume that donors want external evaluation that fits within certain Western-led criteria of quality and rigor (Kelly, 2019, 2021a, 2021b); however, there is a paucity of evidence exploring these assumptions.
As mentioned in the introduction, the first author (LK), an Australian-based external evaluator, agreed to start the evaluation contract with the idea of phasing out and passing over to a Myanmar national as soon as practicable. The total contract involved four evaluations over the three-year program cycle (baseline, two annual formative interims, and a summative follow-up). LK was contracted close to the due date of the baseline evaluation. The first evaluation was conducted in a standard manner with the external evaluator responsible for the design, conducting the in-country fieldwork, extracting the data back to Australia for interpretation, writing up the results and recommendations, and disseminating the report to the NGO and donors.
At the same time, and in line with their commitment toward community-led, co-designed, programming, the NGO was prioritizing an embedded village-level approach to monitoring and evaluation using ActionAid's Village Book methodology (ActionAid, n.d.). This approach guides villagers to record their own baselines, context, plans, priorities, challenges, and successes and acts as a tool for demonstrating local change, which can then be used to motivate themselves and neighboring villages and advocate for government support (Ware, 2018). Training and practice using the Village Books started villagers on a journey of formalized evaluation capacity building, which, according to focus groups with village residents, they have found helpful, inspiring, and effective. Residents note that using the early entries in their Village Books for advocacy purposes has resulted in local government provision of materials and labor for infrastructure projects and teacher salaries for education projects. The Village Books represent a promising methodology for decolonizing evaluation. However, the coup and pandemic meant that communities have been unable to progress their Village Books, meaning that we were unable to utilize this data for the evaluation.
While the Village Books began developing evaluation capacity within villages for internal evaluation, the baseline external evaluation was conducted almost entirely by the Australian-based external evaluator with some local consultation on the design, support collecting the data, and discussion with staff around recommendations. Noting the limited collaboration with locals surrounding this and previous external evaluations, the baseline evaluation outlined that: Ideally, external evaluation would be conducted by someone local to the region. As no one suitable was available, the independent evaluator used the data collection phase for this evaluation to begin training [staff] how to conduct interviews and accurately record interview data. This included discussing confidentiality, ethics, prompting, suspending bias, and noting interesting data outside of the exact question asked. Over the three-year project cycle, it is envisaged that local people will play an increasing role in monitoring and evaluation while the role of foreign-based independent evaluators will reduce (Kelly, 2020, p. 11).
Further, the recommendation section of the baseline evaluation suggested that: it would be worthwhile for [the NGO] to consider working towards an evaluation framework that supports and upskills local people, rather than relying on extractive outside consultants. This could be actioned, as a start, by working towards hiring an internal evaluation staff member in [the country office of the NGO] who is mentored by the independent evaluator. At the very least, it would be beneficial for the independent evaluator to keep this in mind and seek to upskill [NGO] staff and other local people with evaluative skills whenever the opportunity arises (Kelly, 2020, p. 34).
As our awareness grew, we looked back to the baseline evaluation recommendations and saw the trend toward upskilling and capacity building others rather than understanding the nuance of decolonization as two-way skill-sharing and recognition of the knowledge already held locally. In the time between the baseline evaluation and the first interim evaluation, NGO staff and the Australian-based external evaluator planned steps toward decolonizing evaluation by prioritizing local involvement, as stated in the recommendations. Pivotal to this was the need to secure a Myanmar national to conduct the evaluation.
The NGO secured an emerging evaluator from Myanmar (the second author of this paper, PPTH) who was temporarily based in Australia for tertiary education purposes. As mentioned in the introduction, despite Rakhine heritage, PPTH is from a state in Myanmar other than Rakhine; however, she has a strong understanding of Myanmar culture, language, norms, and mores and was raised in a rural area of a neighboring state facing similar challenges to Rakhine. We began collaborating in mid-2020, five months before the fieldwork for the interim evaluation was due to commence. The initial plan was for PPTH to conduct the fieldwork, early analysis, and drafting, with LK providing support and assisting with the final analysis and write-up. However, the COVID-19 pandemic meant that travel to Rakhine State was no longer an option and forced a different plan.
The pandemic workaround had several strengths and limitations. We were unable to gather most of the data required ourselves due to travel restrictions and internet connectivity issues. This necessitated greater utilization of frontline NGO staff and local researchers in data collection from village residents and facilitators and pushed us to consider the next steps toward decolonization. All these data collectors were necessarily Buddhist Rakhine due to Rohingya travel bans, low literacy, and limited ability to speak Rakhine. This has obvious drawbacks, and some potential benefits, that will be discussed later in this paper. Wherever possible, a person who was not directly involved with the program conducted the data capture activities. While there are some concerns around bias and objectivity, the local data collectors were pre-briefed on how to minimize bias and data was sense-checked by us and the Country Director. The people conducting the interviews and focus groups have had minimal research training, and connectivity issues prevented us from effectively supporting these workers. However, one of the local data collectors previously worked closely on the baseline evaluation in 2019 and received some research training at that time. These data collectors have also taken part in reflective dialoguing around anti-Rohingya attitudes and assumptions.
As a result of the pandemic, the interim evaluation was based on the following datasets:
In-depth interviews with in-country NGO staff (n = 3): These interviews were conducted by PPTH over online video conferencing platforms. Interviews with the 12 village facilitators (all Buddhist Rakhine, n = 12): The interviews were conducted face-to-face with the local Rakhine researchers travelling to the villages or the interviewees travelling into Sittwe for discussions. Focus groups with village residents in program villages (n = 84): Focus groups were facilitated face-to-face by the local Rakhine researchers. This caused some issues in the Rohingya focus groups as not all Rohingya people speak Rakhine or Burmese and they had to interpret for one another. In total, 12 gender-segregated focus groups were conducted in ten villages: six groups in five Buddhist Rakhine villages (41 participants) and six groups in five Muslim Rohingya villages (43 participants). The total of 84 village residents who participated in the focus groups comprised of 48 female and 36 male participants. Each focus group ran for approximately 40–55 min and had an average of seven participants.
Data from the interviews with facilitators were provided to the external evaluators as audio files while the results of the focus groups were recorded in handwritten notes, written in Burmese. The recorded interviews were transcribed by PPTH, translated into English, and added to the comprehensive notes taken during the interviews with key in-country NGO staff. Additional clarifying data was collected by PPTH throughout the evaluation via online platforms and telecommunications with frontline staff and facilitators.
PPTH conducted a thematic analysis of the three sets of qualitative data (staff interviews, facilitator interviews, and villager focus groups). This involved close reading and familiarization with the data and then coding the text line-by-line to highlight the topic covered by each string of text (Braun & Clarke, 2006). She then grouped the strings of text with similar codes together to surface key themes. The three datasets were coded individually; however, the same themes were identified. This allowed for cross-dataset comparisons and triangulation, strengthening the trustworthiness of the findings.
PPTH conducted the initial write-up with LK leading the second round of analysis and final write-up. The draft report was circulated to the local Country Director as well as other key staff members to ascertain their input regarding the findings and recommendations. The final report was disseminated to key NGO staff and donors. A poster version of the findings was developed in the Burmese language for dissemination in each participating village.
While we co-led the interim evaluation discussed in this paper, the responsibility for leading the evaluation, and moving the process toward decolonization, has since transitioned to PPTH, with LK's role moving to that of a critical friend. Importantly, we see the role of critical friend as one to offer provocations and help evaluators reflect on their practice. By moving the tasks of the critical friend away from offering methodological or other technical advice, we hope to reduce the influence of Western ways of evaluating and provide space to nurture and promote local ways. In this, the critical friend will continue their own development, with eyes opened to other perspectives.
Reflection on Progress Toward Decolonization of External Evaluation
Journeying toward decolonization is a vastly complex undertaking that seems to expand in its complexity as our awareness grows. Our reflections capture this fluctuation of perceived progress. In our early naivety, we considered that hiring a Myanmar national to lead the evaluation, capacity building locals around Western ways of evaluating, and translating a short excerpt of the findings into Burmese constituted good progress. Obviously, that was highly inaccurate. At best, we are in the very early contemplation stages of implementing a decolonizing approach, noting the common co-option of doing decolonization (Tuck & Yang, 2012).
Also noting this naivety, PPTH reflects on the following: The concept of decolonizing is very new to me. Through my education journey, I have learnt colonized knowledge through a colonized education system dominated by Western knowledge and ideology. This resulted in inability to question, analyze, and think critically. More importantly, learning about research, evaluation methodology, and methods is a privilege in the Myanmar education setting that I did not experience much. Later on, while I was working for community development projects in partnership with international development organizations, I started to realize that voicing opinions, valuing local culture and knowledge, and having an equal power relationship between local community actors and international development workers are essential to effective project outcomes. This understanding began the journey of reflecting on the effects of colonization and the need for decolonization in international development evaluation. Many evaluators and development workers including myself see Western ways of evaluating as problematic (e.g., Eyben et al., 2015; Kelly, 2021b). Obviously, that is not incorrect, but insufficient. The problem is also local evaluators’ inability to question, challenge, and critically analyze Westernized evaluation knowledge and methodologies. This means that decolonization is not the responsibility of foreign evaluators alone; respective countries’ education institutions, organizations, media, and the people of Myanmar as individuals, including myself, all have important roles to play in shifting power relationships and supporting local evaluators’ consciousness-raising that questions and analyzes Western ways of evaluating.
In the following subsections, we unpack our progress so far, using the intersecting concepts of (de)colonization outlined in the literature review to measure our thinking and highlight areas requiring increased attention. We have structured this section according to the three questions we ask in our retrospectives, which are meetings held between the two of us where we discuss what worked well, what could be improved, and decide on the next steps (Kelly & Rogers, 2022).
What Worked Well?
Initially, we felt the evaluation process went well and was an excellent first step on our decolonization journey. As a Myanmar national, PPTH was able to bridge between local and Western ways of being, knowing, and doing, as well as offering a deeper understanding of local culture and subtle contextual nuances. We worked well as a team with LK providing technical research and evaluation mentoring and PPTH providing contextual sensitivity and language skills. It was not until later in the piece, after engaging with the decolonization literature more deeply, that we saw the colonizing nature of our approach whereby Western methodologies were privileged and Myanmar nationals were viewed as passive receptacles receiving top-down knowledge from Western experts.
During the retrospective that we held immediately after completing the evaluation, we agreed that it worked well having PPTH conduct the interviews with staff as she was able to ask follow-up questions and take comprehensive notes. Although COVID-19 travel restrictions meant that these interviews had to be conducted over the internet or telephone, fortunately, the connection to Sittwe worked well and allowed for productive discussions. PPTH's ability to conduct the interviews in Burmese helped gather a greater depth of information than would have been possible if the interviews had been conducted in English with an interpreter. While the interviewees speak Rakhine as their first language, Rakhine and Burmese are similar yet distinct languages with the same written script. Additionally, most Buddhist Rakhine people speak Burmese. Again, after further reflection, we surfaced issues with this approach. Despite this group being well able to speak Burmese, the Burmese colonized the Rakhine. As such, conversing in the Burmese language reinforces colonization. Further, as highlighted elsewhere (e.g., Chilisa, 2019; Kovach, 2010), the interview can be a colonizing method itself.
In terms of fulfilling organizational and donor needs, it worked well having local people collect the majority of the data, PPTH collect the remainder and conduct the initial analysis, and LK lead the final analysis and write up, which was augmented through discussion with local staff. The report satisfied NGO and donor needs, and we broadcast the inclusion of locals in the data collection and analysis. However, the English language reports we produced were completely inaccessible for the community recipients and even for some of the in-country staff, as discussed in the next subsection on required improvements. Further, local inclusion in the analysis and interpretation of the data, and in the development of the recommendations, was inadequate as it mostly only involved the Country Director with some secondary input from other local staff and community facilitators filtered through him.
In a positive step toward decolonization and downward accountability, the poster PPTH created in the Burmese language showcased the key findings and recommendations that were identified through the evaluation. The poster was well-received by local staff and was printed and displayed in each of the villages participating in the program with the village facilitator running a community feedback session to discuss the evaluation findings. The poster was created with low literacy in mind and included several pictures. While this is a good first step, the poster was only provided in the Burmese language and was not translated into Rakhine or Rohingya. There are several issues with this in addition to high illiteracy rates, especially among the Rohingya population who have been denied education. Linking again to Burmese colonization of the Rakhine, another issue surrounds the fact that a large percentage of the Rohingya villagers participating in the program would have been unable to understand the posters without fellow residents interpreting it for them. Further, the Rohingya are oppressed by the Rakhine and the Burmese, so they suffer a multi-layered colonialization that was reinforced by excluding their language and views on appropriate dissemination methods.
What Could We Improve?
There is much that we can improve, as touched on in the section above. Noting that decolonization is a perpetual process, we commit to staying alert for the aspects of colonization that we become cognizant of as our awareness builds. We could improve by unlearning some of the standard rules of evaluation and by challenging deeply ingrained assumptions around evidence, rigor, and credibility (as outlined in Table 1 previously). We could improve by asking, listening, and hearing quiet voices and including them at each stage of the evaluation from planning to dissemination. We could reconceptualize capacity building away from something we do to others into a collaborative process of co-inquiring with community, where learning is reciprocal and local knowledge is prioritized.
While we took a first step toward decolonizing data collection by supporting local data collectors, there were several challenges with this approach. The choice of methods and interview question schedule was devised by the Australian-based external evaluator with minimal additional input. To progress decolonization it would be optimal to conduct collective dialogue with local staff and communities to identify their goals and indicators of success, and then plan out a means of measuring their achievements toward their goals. Instead, the local workforce's main responsibility was organizing logistics and supporting the local data collectors to identify suitable participants for the evaluative research. Additionally, the local researchers were all Buddhist Rakhine, as rationalized earlier in this paper. Alternatives were unavailable at the time; however, utilizing only people of one ethnicity for these tasks is a colonizing act, one that requires effort to devise a solution whereby Rohingya people take the role of local researchers, especially in their own communities. While COVID-19 travel restrictions hindered facilitation of collaborative dialoguing sessions, it would be optimal to have PPTH guide community members, frontline staff, and local Rakhine and Rohingya data collectors in co-creating decolonized methods and formulating locally trusted ways of capturing the results.
Once the data was collected, it was extracted to Australia for an initial round of analysis conducted by PPTH and a secondary round by LK. Again, this was conducted with little input from local stakeholders with only some of the final analysis and recommendation development discussed with local staff. Using a participatory analysis process could have helped identify themes of particular importance to local people and provided a different lens through which to examine the data. The opportunity for this participatory analysis was greatly impacted by PPTH's inability to leave Australia due to COVID-19. Pending the situation with the military coup in Myanmar, as well as ongoing pandemic restrictions, having the external evaluator in-country will enable this participatory approach in the future. It will be important to ensure that any participatory analysis is undertaken with both Rakhine and Rohingya villagers.
Although the draft and final versions of the report were shared with the organization, they were not shared with local facilitators in the villages. As well as a power imbalance that may have meant sharing the reports was not considered, the reports were in English, which posed a significant barrier as the local facilitators, and some of the leadership team in the local office, do not speak English. A lack of access to the draft report meant that input from the majority of local staff at the draft stage was not possible. Only the highly educated local Country Director was able to provide input to shape the final version. It would have been better if PPTH, the local office staff, and the local facilitators had a meeting to reflect on the findings and develop the recommendations for the report. The recommendations should also include the community members’ perspectives. Additionally, the findings and recommendations should be accessible for the organizational workforce and communities, which necessitates provision of the information in the local Rakhine and Rohingya languages. Inclusion in this aspect of the evaluation, through revision of the draft report and development of the recommendations, could help enthuse local staff and community members around evaluation, especially if prior work is undertaken to facilitate development of locally agreed goals and indicators of success. Rather than leaving the recommendations with the leadership team to implement, it would be beneficial if we supported the organization to meet with facilitators and Rakhine and Rohingya village committees after the final report to plan out implementation of the locally generated recommendations.
Similar to the language barriers noted, dissemination of the findings and recommendations is another space for decolonization. The posters, while written in Burmese and created with low literacy in mind, should be written in Rakhine and Rohingya. Additionally, the main reports should be made available to those interested in an appropriate format and language, which may not be in the written report form so cherished by Western mindsets. Knowledge translation and dissemination could be greatly improved by including local staff and Rakhine and Rohingya community members in developing the materials in whatever format suits them. This could involve collaborative sessions where the findings and recommendations are discussed and reworked into decolonized artefacts with meaning and relevance to local people.
Next Steps
As noted above, decolonization requires us to remain cognizant and reflective of our actions at each stage of the evaluation and attempt to redistribute power and remake our roles as facilitators of evaluative thinking rather than external experts. If global and national forces prevent PPTH from conducting the next evaluation in-country, the ability to support decolonization will be reduced as donor compliance needs endure, which may necessitate another evaluation produced with little progress toward decolonization goals. The NGO can identify Rohingya people who could take on the role of local researcher in their community and engage them in critical discussion around appropriate methods and research skills alongside the local Rakhine researchers. However, the ability to effectively co-inquire together will be limited as physical movement between areas is fraught and internet and telephone connectivity is poor, further diminished by the military coup. Local staff and communities can be supported to run data analysis sessions and develop recommendations from the findings, but this will be extremely difficult without a physical presence in-country.
Realistically, if PPTH is unable to travel to Rakhine State for the next evaluation, she may have to focus on the elements of decolonization that she can progress. She can spend time reflecting on her attitudes and assumptions toward Rohingya people and assess the impact of these biases on her evaluative practice. She can suggest that Rohingya people are identified and secured as data collectors. She can work closely with the local staff who have access to the internet to analyze the data and develop recommendations. She can also focus on disseminating the findings and recommendations in a manner that makes them accessible for all community members taking part in this project. This can be achieved by supporting local staff to discuss findings and recommendations with Rakhine and Rohingya village committees and sourcing their ideas and input to produce useful evaluation artefacts.
Discussion
Reflecting on our progress toward decolonized evaluation highlighted some key areas for improvement and outlined next steps in the journey. This section examines these findings against the constructs presented in the literature review to map and critically analyze the extent to which we have begun to decolonize knowledge, culture, politics, power, and evaluation frameworks and practices.
Decolonizing Knowledge and Culture
To decolonize knowledge and culture in evaluation, it could be helpful for the NGO staff and community recipients to engage in co-inquiry to understand the evaluation process and why evaluation is needed, how evaluation can be used for organizational development, and how evaluations can be produced together. However, the way evaluation has been conducted to date has predominately framed evaluation around upward compliance instead of learning. This is not the fault of donors, evaluators, or organizational executives, but the internalized notions of evaluation perpetuated within the Western discipline (Kelly, 2021a). While we have sought to bring staff and communities along on the journey, this has been a tokenistic involvement with significant room for improvement and greater inclusion of participants, particularly in the analysis, data interpretation, and recommendation generation phases. The interim evaluation that is the focus of this paper has started positive steps toward decolonization of data collection, but the other phases of evaluation, including planning and reporting, are led externally.
This highlights an important next step for us—to work alongside staff and community recipients to reframe evaluation as a learning process that can help them achieve their goals and help the organization maintain accountability to the community they serve, as well as donors. Part of this process would involve supporting local staff and communities to create their own evaluative methods and processes rather than upskilling local people with Western ways. Importantly, decolonizing evaluation will require that local knowledge and cultural values are shared and promoted across Rakhine and Rohingya communities through consciousness-raising processes that help surface constructed knowledge and culture in both communities.
Decolonizing Politics and Power
To decolonize power and politics in evaluation, it is import to surface and adjust the power relationships between evaluators, donors, NGOs, and community recipients. The people of Myanmar have endured many decades of oppression by the government and military. PPTH notes that this means people are not used to questioning, challenging power relationships, or critically analyzing. She feels that people are afraid to raise questions and avoid speaking up even when they disagree. The difficulty of decolonizing power and politics in evaluation is made even harder as the local community appear to fear voicing their opinions. How can we work toward creating equal power relationships when local staff and community recipients are uncomfortable disagreeing with the procedures and approaches of the external evaluators? This is a challenge of the decolonization process in Myanmar in general. However, as mentioned above, the Rakhine experienced multiple colonizations by the British and the Burmese. As for the Rohingya, they have been oppressed by the Rakhine and the Burmese. This highlights the complexity of the power dynamics between the Rakhine and Rohingya, where the Rakhine are oppressed by the Burmese and are the oppressors of the Rohingya. Decolonization of the Rakhine and Rohingya communities discussed in this paper is, therefore, fraught and layered with complexity. The program has implemented community-level feedback mechanisms and provides space to develop critical analysis skills within each village, which could be utilized as sites to discuss evaluative questions, ideas, and concerns.
Decolonization is on an ongoing journey and requires the support of evaluators, community recipients, and the organizational workforce in ongoing dialogue and critical reflection. It requires the utilization of a decolonizing framework to ensure our colonized knowledge and practice is transformed. Local evaluators need support from their institutions and donors to adopt a decolonized approach. As such, institutions, donors, and NGOs should promote evaluation that is responsive to locally led, collaborative community participation, allowing local community voices to emerge and construct their own realities. NGOs and evaluators should understand that decolonizing knowledge in evaluation seeks to change unequal power relationships. This necessitates their commitment toward redistribution of power. As well as being key to working toward reparations of harm, this way of working benefits NGOs, evaluators, and other stakeholders by helping to free us from blinkered and bound knowledge traps. It can result in healthier relationships and more streamlined development processes.
Decolonizing Evaluation Frameworks and Practices
Local evaluators should be supported to create their own evaluation, including through reconstruction of evaluation tools, instruments, strategies, theories, and models to ensure relevancy in local settings. As locally educated Myanmar nationals are unlikely to have understanding and expertise around Western methodologies and methods, this privileges foreign evaluators or foreign educated local evaluators who are seen as more proficient and technically sophisticated in the eyes of executives hiring evaluators for international development projects. This situation suggests that Western evaluation methodologies are viewed as best practice while credentials and experience from Western universities and international organizations enhances the desirability of evaluators tendering for jobs. Viewing expertise and proficiency through this blinkered paradigm restricts the pool of qualified and credible evaluators and privileges people who may limit the decolonization of evaluation. Avoiding this trap necessitates an inversion of expectations, where local knowledge, interpersonal skills, and innovation is desired over formal qualifications and standard evaluation expertise.
The guidance provided by community-centered approaches, such as empowerment evaluation (Fetterman & Wandersman, 2005; Fetterman, 2015) and Indigenous evaluation (LaFrance & Nichols, 2009, 2010), offer frameworks to support decolonized practice through addressing power imbalances. These include having local people lead the initiation and design of evaluations, determine culturally relevant and respectful methods for gathering data, and analyze and interpret data through strategies that are aligned with cultural context and that consider intersectionality (Kawakami et al., 2007). Through utilizing bottom-up evaluation techniques, evaluation processes and findings can foster self-determination, ownership, agency, and empowerment (Fetterman, 2015). These approaches align with and support decolonization. They increase the potential for evaluands to achieve results that are meaningful to the people the programs and projects are intended to support by focusing on community recipients as the planners, implementers, and evaluators of their own programs (Wandersman & Snell-Johns, 2005).
Conclusion
This paper has explored how effective we have been at implementing a decolonizing approach to evaluate the case study evaluand in Rakhine State. It found we have a long way to go on this lifetime journey of discovery. Consciously embarking on this process has involved engagement with the decolonization literature, continual reflection on our practice, and introspection. We began with the notion that organizing locals to facilitate data collection and building their capacity around Western evaluation methods was a sufficient start. We quickly realized our reductionist error as the complexity of the decolonization process grew alongside our increasing awareness. The act of writing this paper offered us an opportunity to rethink and deconstruct our deeply held beliefs around what constitutes good evaluative inquiry. We hope that our learnings and provocations support others attempting to decolonize their practice. Further, we hope that this paper helps evaluators question their positionality, consider whether they are best placed to act as evaluators for a given evaluand, and question the supremacy of Western ontologies and epistemologies that underpin dominant narratives of the evaluation discipline (Chouinard, 2016).
As colonization diminishes colonized peoples’ power and identity, evaluation can play a role in undoing that harm by prioritizing self-determination and acknowledging people's strengths and capabilities that were suppressed and repressed by colonization. As Tuhiwai Smith (2021) outlines, decolonization is about seeing colonized people as fully human and returning power taken from them, not as an act of altruism, but one of restitution for misdeeds. Supporting and facilitating people to evaluate progress toward their own goals, against their own measures of success, reinforces their humanity and agency. At the same time, the shift in power helps colonizers become more fully human by reinstating equilibrium and moving them from a role of close-minded leadership to one of continual learning through listening, understanding, and being open to other ways of being, knowing, and doing.
Footnotes
Acknowledgments
The authors would like to thank Sarah Williams and the anonymous peer reviewers for their thoughtful provocations and encouragement.
Author's Note
Phyo Pyae Thida (aka Sophia) Htwe, University of Melbourne, Melbourne, Victoria, Australia.
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.
