Abstract
This article describes our experience of conducting a 5-year, culturally responsive evaluation of a federal program with Indigenous communities. It describes how we adapted tenets from “participatory evaluation models” to ensure cultural relevance and empowerment. We provide recommendations for evaluators engaged in similar efforts. The evaluation included
Keywords
Introduction to Older Americans Act (OAA)/Title VI
Title VI of the OAA supports the well-being of American Indian, Alaska Native, and Native Hawaiian (AI/AN/NH) elders by providing nutrition and supportive services. Such services include congregate and home-delivered meals, information referral, transportation, and caregiver support for individuals providing care for elders or for elders raising grandchildren or caring for adults with disabilities. These services help reduce the need for costly institutional care or medical interventions and represent an important part of AI/AN/NH communities’ comprehensive health and social services. The Administration for Community Living (ACL) within the Administration on Aging (AoA) administers the Title VI programs. In 2017, ACL awarded more than 270 grants to tribes, tribal organizations, and organizations serving Native Hawaiian elders to support local Title VI programs over 3 years. ACL contracted ICF, a research and evaluation consulting firm, to conduct a comprehensive evaluation of ACL’s AI/AN/NH Title VI grant programs.
The evaluation team and authors represent diverse cultural backgrounds and perspectives including Black, Indigenous, Latinx, and White ethnicities. Each evaluator has over 10 years of experience conducting evaluations and providing program or evaluation training and technical assistance (TTA) in AI/AN/NH communities. This article shares our experiences conducting a culturally responsive and participatory evaluation of the ACL Title VI programs. We discuss how we adapted recommendations from participatory and Indigenous evaluation models to conduct an evaluation of a federal program while prioritizing cultural relevance and empowerment of Indigenous communities. We close with recommendations including culturally responsive strategies that evaluators can bring to their future work with Indigenous communities.
Culturally Responsive Evaluation
Culture can be defined as the shared experiences of people, including their languages, values, customs, beliefs, and mores. It also includes worldviews, ways of knowing, and ways of communicating. Culturally significant factors encompass, but are not limited to, race/ethnicity, religion, social class, language, disability, sexual orientation, age, and gender. Contextual dimensions such as geographic region and socioeconomic circumstances are also essential to shaping culture. (American Evaluation Association [AEA], 2011, p. 2)
Culturally responsive and Indigenous evaluations provide a framework for designing and conducting an evaluation in a way that recognizes culturally defined ethics, norms, values, beliefs, and traditions (Bowman et al., 2015; Chouinard & Cousins, 2007; Frierson et al., 2002; LaFrance, 2004; LaFrance et al., 2012; Hood et al., 2005 , 2015; Martinez et al., 2018; SenGupta et al., 2004; Symonette, 2004). While following the fundamental stages of evaluation as described in the Framework for Program Evaluation in Public Health from the Centers for Disease Control and Prevention (CDC, 1999), a culturally responsive evaluation emphasizes a culturally grounded understanding of the community and programs under study (Hood et al., 2015). Many evaluators have noted that an evaluation takes place within the context of human experience, including social, cultural, and historical experience—thus, in some respects, all evaluations reflect culturally influenced norms and values (AEA, 2011; LaFrance, 2004; SenGupta et al., 2004). However, a culturally responsive evaluation places intentional and explicit focus on culture as a part of the evaluation design and implementation with the goal of ensuring ethical, high-quality, and relevant evaluation and to strengthen the responsiveness, validity, and utility of the evaluation findings. Such an approach prioritizes inclusiveness, particularly in the case of populations and communities that historically have been marginalized, seeking to “bring balance and equity into the evaluation process” (Hood et al., 2015, p. 283). In addition to this focus, an Indigenous evaluation recognizes tribal sovereignty, ensuring tribal governments have oversight of any evaluation.
Indigenous Cultural Context and Considerations
Compared with other populations in the United States, AI/AN populations experience significant health and socioeconomic disparities, including a higher prevalence of many chronic conditions (Arias et al., 2014; Espey et al., 2014). In addition, across minority populations, including AI/AN/NH, older adults face various challenges to health and well-being such as reduced access to health care, low socioeconomic status, and higher disease burden compared with non-Hispanic older White adults (Adakai et al., 2018; CDC, 2013). Within the United States, the AI/AN population has the highest rate of disabilities (CDC, 2008; Goins et al., 2007), and the average life expectancy among AI/AN populations is 5.5 years lower than the country’s all races population (73.0–78.5 years, respectively; Indian Health Service, 2019).
A program evaluation can support health and well-being in Indigenous communities by providing data and feedback to inform local programs and guide adjustments to better meet specific challenges faced by the community. However, some researchers and evaluators have betrayed Indigenous communities’ trust by failing to conduct research in an inclusive, collaborative, and transparent manner. “Helicopter research”—wherein outside researchers impose their own research agendas on Indigenous communities without taking the time to build relationships, establish trust, and gain an understanding of community needs, context, and cultural nuance—have left many Indigenous communities understandably wary of outside research (Goins et al., 2011; LaFrance, 2004; LaVeaux & Christopher, 2009; Martinez et al., 2018). At times, intensive scrutiny from outsiders coming into Indigenous communities has “brought little more than loss of cultural ownership and exploitation to Indian people” (LaFrance, 2004, p. 40), and some communities have experienced “deficit-based evaluations” (Martinez et al., 2018, p. 33). In addition to characterizing Indigenous communities in negative and inaccurate ways, in some cases, these approaches to research and evaluation neglect the wealth of experience, skills, and interest of community members to engage in evaluation and do not expand the capacity of community members to continue their own inquiry.
Given this context, an ongoing challenge for evaluators working in these communities is “to move past ingrained reticence toward research and instead actively engage the key stakeholders in creating the knowledge needed to deliver effective services” (LaFrance, 2004, p. 41). In direct contrast to the disempowering experiences some Indigenous communities have had with research and evaluation, a culturally responsive evaluation offers a “welcoming space where evaluators and evaluations honor the strengths, respect the diversity, and authentically include, engage, and empower evaluators and the communities they are working with (not ‘on’) in the evaluative process” (Bowman-Farrell, 2018, p. 552). Thus, central to a culturally responsive evaluation is a focus—first and foremost—on meaningful engagement with the community.
From CBPR to Indigenous Evaluation Frameworks (IEFs)
Meaningful stakeholder engagement approaches, such as collaborative, participatory, and empowerment evaluation frameworks, offer some of the tools and strategies needed to implement a culturally responsive evaluation in Indigenous communities. Emphasizing “a profound respect for people,” these approaches are fundamentally designed to “address concerns about relevance, trust and use in evaluation” (Fetterman et al., 2018, p. 1). Stakeholder engagement can also build capacity, knowledge, and awareness of the benefits of evaluation by inviting and supporting participation of multiple stakeholders in the evaluation process (Fetterman et al., 2018). Among such stakeholder engagement approaches, CBPR and participatory action research (PAR), in particular, have been widely viewed as offering key strategies for carrying out research and evaluation in Indigenous communities in a respectful manner (Belone et al., 2016; Boyer et al., 2005; Caxaj, 2015; Fisher & Ball, 2003; Israel et al., 1998; Sahota, 2010). These approaches focus on building resources within communities, facilitating collaborative partnerships, and promoting a “co-learning and empowering process that attends to social inequities” (Israel et al., 1998, p. 179).
While CBPR and PAR have been widely used in Indigenous communities, they do not necessarily emphasize the context of Indigenous communities. The IEF, Culturally Responsive Indigenous Evaluation (CRIE), Tribal Participatory Research (TPR) and Tribally driven Participatory Research are examples of culturally responsive approaches that build on the principles of CBPR and PAR to describe key considerations and recommendations for conducting evaluation in Indigenous communities (Bowman et al., 2015; Bowman-Farrell, 2018; Bowman-Farrell & Dodge-Francis, 2018; Fisher & Ball, 2003; LaFrance et al., 2012; Mariella et al., 2009; Sahota, 2010). These recommendations include attention to data sovereignty (the right of Indigenous communities to govern the collection, ownership, and use of their data), a participatory ethic, and a focus on partnership building. IEF, CRE, and TPR further recommend that evaluators recognize and emphasize Indigenous and community-specific values, traditions, and ways of knowing as central to evaluation practice. For example, an Indigenous evaluation honors tribal sovereignty and self-determination by ensuring tribal governments review and approve all study protocols including those related to data use and dissemination. An Indigenous evaluation also integrates opportunities to build evaluation capacity within the community itself whenever possible. Participatory workshops with Indigenous partners to build a conceptual model and evaluation plan for their program “demystifies the process of evaluation and builds ownership in the evaluation” (LaFrance, 2004, p. 46). As described by LaFrance et al. (2012, p. 59), context “grounds all aspects of Indigenous evaluation…. programs are understood within their relationship to place, setting, and community, and evaluations are planned, undertaken, and validated in relation to cultural context.” In addition, an evaluation in an Indigenous context should promote Indigenous ways of knowing which are often excluded from Western evaluation models (Bowman et al., 2015; Bowman-Farrell, 2018; Bowman-Farrell & Dodge-Francis, 2018; LaFrance et al., 2012; LaFrance, 2004).
In designing and implementing the Evaluation of the ACL Title VI Programs, we considered recommendations and best practices from a wide range of sources, including the participatory models noted, such as CBPR and PAR, and Indigenous evaluation approaches, such as IEF, CRIE, and TPR, to design an evaluation that would be methodologically rigorous—to meet the needs of a federal program evaluation—as well as culturally relevant and responsive to the needs of the participating Indigenous communities. Specifically, drawing on these approaches, we focused on four core components as important to developing a culturally responsive evaluation in an Indigenous context: (1) stakeholder engagement and a participatory ethic, (2) attention to cultural values and Indigenous ways of knowing, (3) recognition of and respect for tribal sovereignty, and (4) training to build local evaluation capacity.
Methods: Promoting a Culturally Responsive Approach
Forty years after the federal government established Title VI under the OAA, the Evaluation of the ACL Title VI Programs was an opportunity to understand the value and impact of the Title VI programs for elders, their families, and their communities. We used a mixed-methods evaluation approach including two studies: an implementation study to understand the context of the Title VI programs, including barriers and facilitators to program implementation, and an outcomes study to assess the overall effects of the programs on elder health and well-being.
ACL Title VI Evaluability Assessment
Prior to the launch of the evaluation, ACL contracted ICF to conduct an evaluability assessment of the Title VI programs. The evaluability assessment had two goals: (1) to describe the characteristics, context, activities, processes, implementation, and intended outcomes of the Title VI nutrition, supportive, and caregiver support services and (2) to assess the feasibility of, and best approaches for, conducting a full-scale evaluation of the Title VI programs (ACL, 2015). As part of this phase of evaluation preparation, ACL and ICF convened a 20-member stakeholder advisory group with representation from federal and tribal governments and other local Indigenous organizations who could advise on an approach for conducting a culturally responsive evaluation. This advisory group became the Steering Committee for the evaluation.
Evaluation of the ACL Title VI Programs
Building on the results of the evaluability assessment, in September 2016, ICF and ACL launched the Evaluation of the ACL Title VI Programs. The first year of the project (2016–2017) focused on evaluation planning, including the development of the evaluation design and data collection tools and the recruitment of Title VI grantees to participate in the evaluation. Over the subsequent three years (2017–2020), the evaluation team focused on two major annual activities: (1) evaluation data collection, analysis, and reporting interim findings and (2) evaluation TTA for the Title VI grantees participating in the evaluation. The evaluation will conclude in 2021 with a final report summarizing evaluation findings.
In the following sections, we highlight specific strategies that we implemented to ensure a culturally responsive approach—participatory, community driven, and empowering—and to strengthen the responsiveness, validity, and utility of the evaluation for ACL and other Title VI program stakeholders.
Stakeholder Engagement
To develop a culturally responsive evaluation in close collaboration with Indigenous communities, we relied on the key principles of PAR and CBPR, as well as those outlined in IEF, CRIE, and TPR. We knew stakeholder engagement may be challenging as a result of potential community distrust of evaluation, yet the insights of Indigenous program staff and recipients would be critical to ground the evaluation in cultural context and to produce a utilization-focused evaluation design (ACL, 2015). Throughout the evaluability assessment and full-scale evaluation, we worked with Title VI program stakeholders to design and implement the evaluation, including the development and pilot testing of data collection instruments and approaches, participant recruitment techniques, interpretation and analysis of data, and dissemination of evaluation findings. These stakeholders included a wide array of individuals from local, state, and federal agencies that collaborate with the ACL Title VI programs, including local Indigenous program staff and participants. For this evaluation, these individuals and agencies were represented primarily by two stakeholder groups: the ACL Title VI Evaluation Steering Committee and the Evaluation Working Group.
ACL Title VI evaluation steering committee
We convened a 20-member group of federal, state, and local Indigenous representatives to ensure active involvement of stakeholders in planning the evaluation. Recognizing that an Indigenous evaluation approach involves consulting with cultural experts to understand Indigenous ways of knowing for each community (LaFrance et al., 2012), we viewed the creation of this group as fundamental to our approach to reflecting and engaging a variety of stakeholders representing program staff; regional, state, and federal agency partners; and funders (Fetterman et al., 2018; LaVeaux & Christopher, 2009). The Steering Committee provided input and guidance during all phases of the project, beginning with the evaluation questions that mattered to them (i.e., what was important to learn through evaluation): How do tribes operate their Title VI programs? What is the effect of Title VI programs on elders in the community? Are there differences nationally or by tribe or tribal groups? Do Title VI programs that rely only on Title VI funds have different outcomes than do programs that have money from other programs or agencies?
The Steering Committee provided specific recommendations to minimize the burden of the evaluation on Indigenous elders, caregivers, and other participants in the evaluation (e.g., using secondary data sources when appropriate to streamline data collection) and helped inform our strategies for recruiting and engaging other stakeholders and evaluation participants.
Evaluation working group
With guidance from the Steering Committee, we created a process for identifying and recruiting Title VI program grantees to participate in the evaluation. Ultimately, 12 Title VI grantees representing four Title VI regions and six states participated in the evaluation. These grantees, hereafter referred to as the “evaluation grantees,” each identified two to three Indigenous staff members from their Title VI program to serve on a new stakeholder group—the Evaluation Working Group—including the local Title VI program director and one or two designated local community researcher(s).
Throughout the evaluation, we met regularly with the Steering Committee and Evaluation Working Group to share updates on the evaluation and to seek guidance on next steps. We consulted these advisory groups on many issues, including appropriate methods for recruiting elders and caregivers for focus groups, format and content for evaluation TTA for grantees, local data collection, interpretation of data, and approaches for developing evaluation reports and disseminating findings.
Attention to Indigenous Cultural Values
To ground the evaluation in the tribal community, a culturally responsive evaluator should learn as much as possible about its history, resources, governance, and composition. If possible, he or she should engage in community activities such as graduation ceremonies and dinners for the elders in the tribe, or funerals for honored tribal members. Engagement can also involve attending special events such as a Treaty Day celebration, powwow or tribal dance, rodeo, or canoe journey. (LaFrance, 2004, p. 48)
ACL Title VI evaluation logic model and medicine wheel
As part of the evaluability assessment, we developed a Title VI program evaluation logic model and corresponding conceptual model in the form of a medicine wheel (see Figure 1). Development of the logic model helped to ensure a shared understanding of the resources and activities intended to meet program goals and the extent of program implementation. Moreover, the process helped us jointly explore, with the Steering Committee, the theory of how program activities support the dignity, self-respect, and cultural identity of tribal and Indigenous elders and communities (ACL, 2015). Because many Indigenous communities do not conceptualize their programming in the linear “cause-and-effect” framework represented by a logic model, we also developed the Title VI program medicine wheel. The circular shape of the medicine wheel, reflecting wholeness and that all aspects of life are interconnected (ACL, 2015), helped to depict both near- and far-term outcomes from the logic model across the traditional quadrants of Indigenous practice: spiritual, mental, emotional, and physical. Each quadrant is housed within the context of community, family, and intergenerational connection—reflecting the importance and interconnectedness of each to the spiritual, mental, emotional, and physical well-being of Indigenous communities and elders and the overall goals of the Title VI programs.
Title VI program evaluation medicine wheel.
The Title VI program medicine wheel was developed: (1) to ensure that the guiding evaluation framework would be anchored in and reflective of the cultures, values, and traditions of Indigenous communities receiving Title VI program funds; (2) to incorporate dimensions of well-being identified by the Indigenous stakeholders, thereby clearly acknowledging their validity; and (3) to operationalize measurement of such dimensions and concepts (ACL, 2015).
By emphasizing the interconnectedness of elders’ well-being with that of family and community, the Title VI program medicine wheel grounded the Evaluation of the ACL Title VI Programs solidly within the framework of an Indigenous worldview (LaFrance & Nichols, 2010). The medicine wheel also helped us think holistically about the different domains that the Title VI programs can affect and how to meaningfully assess, within the context of Indigenous stakeholder perspectives, Title VI outcomes.
Gathering perspectives through storytelling
As recommended by the Steering Committee, the evaluation design included focus groups and interviews, conducted in the community and in person, with Indigenous elders, caregivers, and Title VI program staff as a way to integrate oral storytelling in the evaluation. Focus groups and interviews helped ensure the centrality of community elders’ voices, along with caregivers and program staff, while acknowledging Indigenous traditions related to oral storytelling (Bowman et al., 2015; Datta, 2018; Iseke, 2013). For example, Bowman et al. (2015) described that “[T]raditionally, for Indigenous people, knowledge development, collection, and transfer are primarily oral processes…[C]ulturally responsive evaluation in this context does not privilege the written word but understands that oral traditions in Indigenous contexts are often more sacred, respected, and protected than the written protocols.” LaFrance et al. (2012) also described that “Storytelling and metaphor…serve as methods to anchor indigenous evaluation to symbolic and textual references holding deep ties to the culture of a people and place.” Iseke (2013) noted that oral storytelling provides opportunities to express the experiences of Indigenous people and the sharing of Indigenous knowledge and culture in Indigenous languages if desired. As one local Title VI program director shared, “I think [the elder focus group] was a benefit to the elders themselves, because it made them feel important. It made them feel like their voice was going to be heard”. These methods provided an opportunity for Indigenous community members to tell their unique stories related to the context, challenges, successes, and components of the Title VI programs.
Understanding community-specific cultural traditions
Culturally responsive evaluation emphasizes the need for evaluators to understand the communities and programs that are the focus of the evaluation. The evaluators should take personal responsibility to become educated about the cultural context of the program and the participants. For example, LaFrance (2004, p. 48) noted that time spent in communities is critical in helping “the evaluator understand the context in which he or she is working. It also allows Indians in the community to build relationships with evaluators that are based on friendliness and respectful interest, rather than defined by strict roles and outsider ‘expertise.’” Informed by this guidance, we included annual two- to three-day site visits with the evaluation grantees providing an opportunity for evaluators to meet with community members, understand their perspectives, gain insight about the cultural context and traditions important to each community, and learn how programs function in the community. Prior to the site visits, Indigenous members of our evaluation team conducted internal trainings for our team to build team capacity to engage with the grantees in a culturally respectful way. Trainings included presentations, readings, and facilitated discussions on the principles of cultural humility and the history of research and evaluation in Indigenous communities (National Congress of American Indians [NCAI], 2012; Substance Abuse Mental Health Services Administration, 2009; Waters & Asbill, 2013). In addition, we developed and shared with all team members our site visit protocols, which emphasized respect for and recognition of Indigenous community members as the experts of their communities and experience. To honor this expertise, team members brought to the community small gifts (e.g., tea or coffee) as a token of appreciation and began each visit with a Native land acknowledgment (U.S. Department of Arts and Culture, 2017). In addition, we opened all meetings, including focus groups with program staff, elders, and caregivers, in a culturally appropriate way as defined and guided by the community. For example, we began each day with a song or prayer led by an elder. During the visits, and guided by local community customs, we spent time with Indigenous program staff, elders, and caregivers. Our goal was to interact in open and inviting spaces and to hear local community member perspectives and stories in familiar contexts—for example, within the Elders’ center, in groups, or individually as appropriate based on community preference, and in the languages or relational styles preferred by participants, in some cases accompanied by a translator and/or family member. This approach was designed to contrast with the prior experience of some of the evaluation grantee program staff, one of whom described: “Sometimes when we get visits from our [federal grantors,] they’re usually really quick in and outs, and sometimes they’re a little intimidating”. In collaboration with each evaluation grantee, we considered the unique needs and context of each community, to support elders and other Indigenous community members in feeling comfortable sharing their thoughts and perspectives. During an interview, one program director shared, “Your people were very sensitive and very knowledgeable. But I think the sensitivity part is what made people really comfortable…”.
When appropriate, and with guidance from each evaluation grantee, during the site visits we met with tribal leadership to introduce ourselves and to clarify the purpose of the visit (i.e., working with the community to gain understanding of how the Title VI program is supporting elders). We also participated in or attended local cultural events, such as community festivals, elder game nights, dance ceremonies, and congregate meals with elders. Guided by grantee program staff or representatives, the team toured communities, visiting community landmarks, social and cultural centers, government buildings, schools, and other institutions and learned about community life in general. These activities were invaluable in facilitating introductions to Indigenous community members, in helping to build relationships, and in providing insight about community needs. Showing up and “taking the time to come out,” as one program director explained, and “your ability to really try to learn about our programs and do those site visits…to connect with us and also with our elders was really appreciated”.
Recognizing Tribal Sovereignty
An important aspect of a culturally responsive evaluation is recognition of tribal sovereignty. Most tribes in the United States are federally recognized, with the right to govern their own members and provide oversight for activities occurring on their own land (LaVeaux & Christopher, 2009; NCAI, 2020). In considering an approach to introduce the evaluation and engage Indigenous community members, we respected that evaluators must interact directly with tribal governments to work in Indigenous communities and that sovereignty also gives Indigenous communities the authority to create their own research and evaluation approval policies (LaVeaux & Christopher, 2009; NCAI, 2020). With these points in mind, we worked with the Steering Committee to launch the evaluation, recruit grantee participants, and engage community members using the approaches outlined in the sections below.
Title VI grantee recruitment
Rather than requiring grantees to participate in a federal evaluation, we emphasized community preference, including both optional participation in the evaluation and the opportunity for community partnership and contribution. In the spring of 2017, we released a request for proposals (RFPs) to Title VI grantees, inviting participation in the evaluation. Written with guidance from the Steering Committee, the RFP included an easy-to-complete form for grantees interested in participating in this national evaluation. Recognizing the importance of tribal approval for participation in any evaluation, the application also requested an initial letter of support from grantees’ tribal governing council chair or administrator.
In addition, while Indigenous communities have control of their own government and policy affairs, some have limited resources, which “can be an important aspect of agreeing to partner with outside researchers” (LaVeaux & Christopher, 2009, p. 8). To remove this potential barrier to participation, we offered stipends for evaluation grantees that could be used in flexible ways to support their participation.
Tribal resolutions and data ownership
In addition to the initial letters of support from the local governing body, we worked with each of the evaluation grantees to identify and obtain the local-level approvals necessary for participation in the evaluation through the tribal institutional review board, institution at large, or other governing or advisory body such as the tribal council. We attempted to keep in mind the history of research and evaluation in Indigenous communities, which has often led to “general distrust of outsiders who come to study, ask questions, and publish their findings” (LaFrance, 2004, p. 40, citing Crazy Bull, 1997). We provided background documents for tribal governing councils that described the evaluation, including the purpose and scope of the evaluation, the expectations and benefits to each stakeholder body (i.e., ACL, ICF, the Steering Committee, the Evaluation Working Group, and the local Title VI programs), and ownership of and access to data. Recognizing that “tribal sovereignty also fuels concern about access to data and uses of evaluation information” (LaFrance, 2004, p. 40), we offered a virtual presentation for any grantee’s governing council to introduce the evaluation and explain important considerations, including data ownership and how local data would be reported. For example, reports and briefings on the findings associated with this project include only group-level analyses, and no information would be reported or disseminated at the tribal or individual participant level. Each participating community signed a tribal resolution and data use agreement with ICF to ensure transparency related to data management, use and reporting, and understanding of each community’s rights to self-determination. This step was designed to recognize and address Indigenous priorities, policy, and requirements and to respect tribal sovereignty, self-governance, and self-determination. 1
Locally defined and driven evaluation participant recruitment
We asked each evaluation grantee to identify two to three people from their local Title VI program to serve on the Evaluation Working Group throughout the evaluation, including the Title VI program director and one or two other staff members who would serve as community researchers. The community researcher role was an opportunity for local program staff who were interested in learning more about evaluation in general to facilitate and guide local data collection activities. For example, we supported community researchers in leading the identification and recruitment of Indigenous elders and their caregivers. The community researchers, themselves Indigenous members of their communities, shared with us their knowledge and insights, including recommendations for recruitment methods that had meaning in the context of their communities and could be integrated with regular program routines. This included, for example, announcements at regular Title VI program events, flyers placed at congregate meal sites, or check-in phone calls with elders. The community researchers also helped develop recruitment criteria and identify the types of incentives that would be valued by participants (e.g., basket of household supplies, gift cards to local stores). These steps helped ensure cultural grounding and integration of locally defined community safeguards (Bowman et al., 2015). In addition to working with community researchers one on one, we provided training for the Evaluation Working Group on topics such as participant recruitment, incentives, data collection, and an overview of protecting participants’ rights.
Technical and Training Assistance for Capacity Building
Consistent with a culturally responsive evaluation, the Evaluation of ACL Title VI Programs included a substantial investment in evaluation TTA, informed and guided by the evaluation grantees’ feedback related to their needs, interests, and evaluation capacity. We began by conducting an evaluation capacity and needs assessment and then implemented evaluation TTA to support data collection, empower and build local evaluation capacity, and support the use and dissemination of evaluation findings. We refined TTA approaches to meet specific grantee requests and needs, such as requests for hands-on, interactive training; practical tips for integrating evaluative practices into local programming; and user-friendly evaluation materials and tools. The overall framework for evaluation TTA included the following:
User-Friendly Evaluation Tools and Resources to Support Title VI Grantees.
Accessible Evaluation Dissemination Approaches
Providing useful and actionable information is a key element of participatory evaluation. To this end, we shared evaluation updates with the Evaluation Working Group and Steering Committee at multiple points throughout the five-year evaluation via e-letter and webinars. We also produced annual interim reports summarizing emerging themes in evaluation findings for each year. To ensure the utility and accessibility of the reporting approach, we consulted with the Evaluation Working Group and Steering Committee regarding stylistic preferences for reporting and critical content.
In response to feedback from these stakeholder groups, we designed the interim reports to emphasize readability and accessibility. Using data visualization best practices, we produced snapshot-style layouts that aligned key textual information with visualizations to promote comprehension and information retention (Evergreen & Metzner, 2013; Midway, 2020). We also created brief report summaries—based on the longer interim reports—tailored to various stakeholder audiences to promote wider dissemination and use of the findings (Baxter & Braverman, 2004; Hutchinson, 2017). For example, we created a Title VI infographic as well as a summary on a single, oversized page, in large font, to share evaluation findings with elders in grantee communities.
Discussion
We designed and implemented the Evaluation of the ACL Title VI Programs by drawing on several evaluation frameworks and theories. These included stakeholder engagement approaches (Fetterman et al., 2018) such as participatory or partnership evaluation, including CBPR and PAR (Israel et al., 1998), and collaborative and empowerment evaluation (Fetterman et al., 2010), as well as Indigenous evaluation approaches such as IEF, CRIE, and TPR (Bowman et al., 2015; Bowman-Farrell, 2018; Fisher & Ball, 2003; LaFrance et al., 2012). We considered that a culturally competent evaluator “draws upon a wide range of evaluation theories and methods to design and carry out an evaluation that is optimally matched to the context” reflecting “the diverse values and perspectives of key stakeholder groups” (AEA, 2011, p. 3) as described in the AEA’s statement on cultural competence in evaluation. We also considered, as described by Martinez et al. (2018, p. 36), “the many calls in the literature to abandon a one-size-fits-all approach to American Indian community-based evaluation” and the importance of “a strong community-driven process.” Considering multiple evaluation frameworks while creating an approach tailored to this project allowed us to meet two goals: to develop and implement an evaluation that was sufficiently rigorous to meet the needs of a federal program evaluation and to work in partnership with the evaluation grantees, on their terms, respecting Indigenous knowledge and insight.
Translating theories or frameworks into practice to build an empowering, participatory, and culturally relevant evaluation required an investment of time and resources that is not always available to evaluators, particularly in the context of a federally funded evaluation. As noted by Whitesell et al. (2018, p. 46), “The full realization of CBPR methods is rare in research with American Indian and Alaska Native communities, particularly given the constraints of funding priorities and grant timelines…” However, these authors also note, and we fully agree, “it represents an ideal that guides research in partnership” between researchers and Indigenous communities.
In recognition of this ideal, ACL set a priority for this evaluation, ensuring sufficient resources and a realistic timeline. Adequate resources and time were needed (1) to facilitate close working relationships among the evaluators, Indigenous grantees, and stakeholders; (2) to establish a forum and processes for ongoing communication and feedback among all stakeholders; and (3) to develop a robust and multifaceted approach including training that helped to build local evaluation capacity for grantees. The timeline also allowed the evaluators to respond in meaningful ways to the ideas and suggestions provided by stakeholders and to ensure that these contributions actually helped to shape and guide the evaluation and training approach as the evaluation progressed. For example, we collected grantees’ feedback about the content and format of the annual in-person trainings for the Evaluation Working Group as part of a continuous quality improvement cycle in which ongoing feedback informed the next iteration of trainings and tool or resource development. ACL also anticipated that while time would be needed to build trust, understanding, and communication in any participatory evaluation, some activities may require additional time in Indigenous communities, such as tribal council review of evaluation plans and tribal permissions (LaVeaux & Christopher, 2009), a crucial aspect of recognizing tribal sovereignty.
In the sections that follow, we describe the lessons learned during our experiences, and we offer recommendations that might be useful for others engaged in similar evaluation efforts.
Lessons Learned
Throughout this evaluation, we continuously learned from and with the Indigenous communities and stakeholders we engaged with. The lessons learned through engagement ultimately strengthened our overall approach and kept us accountable to the key principles and values of a culturally responsive and participatory evaluation.
Diversity across Indigenous grantees
Although we anticipated diversity within and across Indigenous communities, including unique cultural and traditional norms, and tried to account for that in our evaluation design, we found we needed to adjust to ensure the relevance and inclusiveness of the evaluation plan across different contexts. The geographic scope and accessibility of local Title VI programs varied widely—for example, some grantees administered multiple program sites while others operated at a single, central site; some were located in geographically remote and service-isolated areas. Grantees also served significantly different numbers of elders/caregivers, and different cultural norms affected participant recruitment approaches. The extent of experience with evaluation, approach to the design and implementation of local Title VI elder programs (e.g., operations, staffing approaches, services offered, support from supplemental resources/partners), and governance and related requirements (e.g., tribal council structures and tribal approval processes) also differed across grantees. We attempted to address the challenge of those differences by emphasizing flexibility in the evaluation design at the outset. With guidance from the Steering Committee and Evaluation Working Group, we provided several options for qualitative data collection to support program staff when engaging elder participants inclusively in a specific community. For example, while we originally planned to conduct one elder focus group per evaluation grantee, for grantees with multiple program sites, we provided the option of conducting several small group interviews, one per site, to ensure representation of the perspective and voice of elders from those sites. We also developed site visit materials tailored for each evaluation grantee based on our understanding of each grantee’s particular circumstances and needs. These strategies enhanced the relevance of the evaluation for the grantees. As one program director shared:
Actually, I think the product that we saw as a result of our involvements with all of the technical work you put into the charting and the graphics and everything that you did was very clear. And it related precisely to everything that we had voiced as needs and concerns…
Frequent turnover in program staff/leadership
Evaluation grantees experienced frequent turnover among Title VI program staff and stakeholders. When turnover occurred among the local Title VI project staff and leadership (such as the Title VI project director or community researcher), this resulted in loss of knowledge and insight about that community’s participation and engagement in the evaluation. In addition, many of the tribal communities engaged in the evaluation have an annual election cycle in which the tribe elects a new governor or leader each year. Although expected, this turnover challenged us to find ways to ensure consistent understanding of and support for the evaluation among tribal leadership over the 5-year period of the evaluation. Our emphasis on readability and accessibility in our interim reports and related resources, which included brief, “at-a-glance” report summaries, was helpful in briefing new program staff or tribal leaders to the overall purpose, utility, and importance of the evaluation. Secondly, by including two representatives from each grantee on the Evaluation Working Group, we helped minimize the impact of staff turnover.
Interpreting the title VI program logic
As described above, with input and guidance from the Steering Committee as part of the evaluability assessment, we developed a Title VI program logic model. Initially, when asked whether the logic model, which represented all of the program elements specified in official descriptions of the Title VI program, accurately reflects the program as they experience it, some Indigenous stakeholders observed that the construct of spirituality and spiritual well-being was not explicitly represented (ACL, 2015). In response, we explored this concept to understand where and how it could be integrated in the program, which could lead to improved community understanding and outcomes. The result was the creation of the medicine wheel (Figure 1), a graphical conceptual model that integrated this construct.
Ongoing feedback loop: Evaluation TTA
The continuous quality improvement cycle described above—in which ongoing feedback informed the next iteration of trainings and tool or resource development—resulted in increasingly well-received evaluation TTA approaches. For example, following the first Evaluation Working Group in-person meeting (in 2017), after evaluation grantees expressed a need for more hands-on activities/trainings, we revised the format of the second in-person meeting (in 2018) to include interactive activities and adult learning techniques (e.g., lecture/slides, games, hands-on practice, individual and group activities, small group discussion, and handouts). Grantees expressed appreciation for revised training format. As one grantee shared, “The discussion that we had, where everyone shared what we do, gave great ideas and information”. Grantees also shared recommendations for further improvement, particularly increased time for group activity and discussion: “I feel that we need a whole day as opposed to a half day because of the discussions and sharing”. The evaluation team further revised the training approach for the third in-person meeting in 2019, and grantee feedback was positive—“I’ve never been to a conference/training where real tools were provided that can be applied immediately”. Grantees also valued the strong working relationships that had developed across the evaluation partners through such in-person meetings and other collaborative activities. As one program director stated:
I really liked what we had done together. It was work between you and us. And I’ve really liked everything that we’ve done. The things that I have learned and the tools that I was able to use to develop to gain that information from the elders you know. And other areas, caregivers. So I think it’s all appreciation for everything that you all have done.
A realistic timeline
We expected that this evaluation would be time intensive, and it was. We found, however, that providing a flexible schedule for the activities provided opportunities to strengthen the evaluation’s inclusiveness. We set wide time windows in the schedule for each evaluation task (design, data collection, analysis, and reporting) and anticipated the need to adjust. For example, to facilitate recruitment of participants in qualitative data collection, we worked closely with each evaluation grantee as they identified and recruited participants and considered appropriate incentives. The time needed to secure tribal council permissions and to identify and recruit participants varied across evaluation grantees, and several faced delays related to unanticipated events, such as illness among evaluation participants. A flexible overarching evaluation timeline successfully allowed all evaluation partners to contribute and participate according to their community-specific timelines.
Recommendations
For the Evaluation of the ACL Title VI Programs, we drew from a wide range of evaluation frameworks and principles to create a culturally responsive evaluation that addresses the context of the Title VI programs. Collectively, our experiences conducting the evaluation provided valuable insights and lessons applicable to a broad array of evaluation contexts. To evaluators engaged in similar evaluation efforts, we offer the following recommendations and practical strategies for developing a culturally responsive evaluation that facilitates stakeholder engagement, strengthens inclusion and cultural relevance, and enhances evaluation TTA to build capacity (Table 2). While some of these recommendations can require a significant commitment of time and resources, most can be incorporated within the generally limited evaluation resource landscape. In addition, these recommendations have broad applicability across the evaluation life cycle (e.g., pre-evaluation, implementation, and dissemination).
Strategies and Recommendations for a Culturally Relevant Evaluation.
Summary
This evaluation—implemented over five years, from 2016 to 2021, and involving hundreds of evaluation stakeholders and multiple legal jurisdictions, including federal and tribal governments—provides an important example of how relevant evaluation principles and frameworks published in journals and other sources can be translated and implemented to create a culturally responsive evaluation. ACL prioritized a timeline that ensured adequate time to meaningfully engage key stakeholders who, in turn, provided critical insight and guidance to ensure the evaluation was culturally grounded. We prioritized ongoing communication with the evaluation grantees, creating a feedback loop to continuously improve our work and to ensure we were meeting grantee TTA needs. This created a space for collaboration that facilitated understanding grantee needs related to evaluation and data use. Further, the evaluation opened a fertile space for a grantee-led idea exchange that has encouraged ACL to work to develop more tools and resources to better support grantees in its work to serve its communities. We hope our experience and lessons learned during the evaluation design and implementation phases may be useful for other evaluators looking to develop a culturally responsive evaluation that facilitates stakeholder engagement, strengthens inclusion and cultural relevance, and enhances evaluation TTA to build capacity.
Footnotes
Authors’ Note
The content presented here is solely the responsibility of the authors and does not necessarily represent the official views of ACL.
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: The Evaluation of the Administration for Community Living’s (ACL) American Indian, Alaska Native, and Native Hawaiian (Older Americans Act, Title VI) Title VI Grant Program was funded under ACL contract #HHSP233201750007A.
