Advancing an ethical imperative for collaborative approaches to evaluation with low incidence and underserved communities: Insights from a DeafBlind Support Services pilot program evaluation

Abstract

This practice article advances an ethical imperative for guiding collaborative approaches to evaluation with low incidence and underserved communities. We describe our evaluation processes for building transformative partnerships during a DeafBlind Support Services pilot program evaluation in Western Canada using collaborative approaches to evaluation. This program was designed to enable adults living with dual sensory loss to experience self-determined activities in the community that enhance resilience and belonging. Key features involve specialised Support Service Providers who assist DeafBlind community members with accessing and participating in essential community services and social, vocational, educational, and recreational activities. In presenting our co-generated evaluation insights, we advance an ethical imperative describing philosophical foundations, pragmatic actions, and political leadership for implementation of collaborative approaches to evaluation with a low incidence and underserved community.

What we already know:

• Low incidence and underserved communities are comprised of individuals who experience disabilities that occur in less than one percent of the general population and often lack adequate access to healthcare and community services. DeafBlind individuals are members of a low incidence community.

• Collaborative approaches to evaluation are guided by eight principles that inform evaluation processes and require ongoing dialogue and mutual learning to create transformative partnerships and transformational outcomes.

The original contribution the article makes to theory and/or practice:

• The current study describes insights from a DeafBlind Support Services program evaluation and provides a linguistically and culturally unique evaluation context in which to advance an ethical imperative for collaborative approaches to evaluation implementation.

• Through collaborative approaches to evaluation, four key evaluation outcomes were co-generated related to health, engagement, independence, and employment of Deaf and DeafBlind individuals. Another unexpected transformational outcome was the elevation of the voice and visibility of the DeafBlind community.

• We advance an ethical imperative describing philosophical foundations, pragmatic actions, and political leadership for the implementation of collaborative approaches to evaluation with a low incidence and underserved community.

Introduction

This practice article advances an ethical imperative for guiding collaborative approaches to evaluation with low incidence and underserved communities. Successful evaluation outcomes are often associated with building transformative partnerships with those involved, yet the application of Cousins et al.’s (2020) eight principles guiding collaborative practice in evaluation (see section ‘methods guiding implementation of collaborative approaches to evaluation’ for description) with low incidence and underserved communities remains to be examined. Collaborative approaches to evaluation are defined as 'the joint engagement of evaluators working with non-evaluators in planning, implementing, and disseminating evaluation' (Cousins et al., 2020, p. 4). Evaluators operating from positions of privilege employ collaborative approaches to evaluation to empower communities through transformative evaluation partnerships. Engaging in continuous dialogue and mutual learning create transformative partnerships among, and transformational outcomes for, those involved (Shulha et al., 2016). Evidence of transformative partnerships can take many forms and transformational outcomes can be easily missed – hence the contribution of this study to advance an ethical imperative for collaborative approaches to evaluation with low incidence and underserved communities. We argue that the principles guiding collaborative approaches to evaluation offer essential conceptual tools for evaluators and that further illustrative examples describing its implementation processes would benefit evaluators working within culturally and linguistically unique contexts.

Illustrative examples of collaborative approaches to evaluation involving low incidence and underserved communities are needed to guide evaluators (Searle et al., 2020). Low incidence is a term used to describe disabilities that occur in less than one percent of the general population (Canadian DeafBlind Association, 2020). DeafBlind, Deaf or hard of hearing, and blind or low vision communities are low incidence populations that require specialised supports to have equivalent access to educational, vocational, recreational, and social opportunities. Communities requiring specialised support report inadequate availability of and access to resources; thus, these communities remain underserved (Addorisio et al., 2022).

The current study describing insights from an evaluation of the DeafBlind Support Services program provides a unique evaluation setting in which to advance an ethical imperative for collaborative approaches to evaluation for the following reasons: First, the DeafBlind community is culturally and linguistically unique and their accessible language needs are not typically recognised by evaluators (Anderson et al., 2023); second, the commitment of the evaluation stakeholders to collaborative approaches to evaluation and building transformative partnerships; and third, the expertise and experience of the evaluation team members for implementing collaborative approaches to evaluation under conditions of severe time, financial, and human resource constraints. The conceptual ethical evaluation framework of Chouinard and Cousins (2021), which highlights the philosophical, pragmatic, and political applications of ethical evaluation, guided our decision-making related to stakeholder participation, evaluation design, and the dissemination of program outcomes. Mertens and Catsambas’s (2021) descriptions of transformative ethical evaluation practices prompted the evaluators to acknowledge and respect the culturally and linguistically unique context throughout the evaluation. Our description of the evaluation implementation provides insights into building transformative partnerships and the transformational outcomes that arose from the implementation of the eight principles of collaborative approaches to evaluation with a low incidence, underserved community.

Evaluation background and context

DeafBlind Support Services aims to enhance and improve the provision of individualised, strength-based support to DeafBlind individuals seeking opportunities to access essential services along with vocational, recreational, educational, and social experiences with the same frequency, autonomy, and dignity afforded other adults who live in the community. Connect Society and DeafBlind Support Services were situated in the context of a culturally and linguistically unique low incidence and underserved community. The evaluation team brings more than two decades of expertise and experience engaging stakeholders in participatory and collaborative approaches to evaluation within community and clinical settings with a focus on equitable access to evidence-based evaluation processes. The features of the evaluand and partners created a unique context.

It is estimated that 1 in 3000 Canadians are DeafBlind (Canadian DeafBlind Association, 2020). DeafBlindness is a unique disability of comorbid hearing and vision loss severe enough to impact communication and access to information and the environment. The loss of sensory systems required for expressive and receptive communication, orientation, mobility, and independent living skills can significantly compromise the personal safety, quality of life, and well-being experienced by individuals living with dual sensory loss (Dammeyer, 2014). Barriers to participation in the community and failures to recognise accessible language needs contribute to the loneliness and isolation of DeafBlind individuals (Kramer, 2014; Hall et al., 2019). Low incidence populations, including members of the DeafBlind community, lack currency and are often underserved by service organisations and unseen by community supports (McDonnall & Cmar, 2019). In the spring of 2019, Connect Society invited the university-based evaluation team to evaluate the impacts of the DeafBlind Support Services pilot program to meet their impending funding requirements.

Provincial government funding of the pilot program was provided to the Alberta Society of the DeafBlind in 2018. The society in turn invited the Connect Society, a non-profit agency that aims to strengthen connections among Deaf, hard of hearing, and hearing people, to design an adapted, person-centred program for DeafBlind adults. DeafBlind Support Services was launched and, in its first year, involved seven Support Service Providers (five Deaf, one hard of hearing, and one hearing) supporting 12 DeafBlind individuals, a Deaf program manager, a DeafBlind booking coordinator, and the hearing Connect Society leader. DeafBlind Support Services seeks to enable adults living with dual sensory loss to experience self-determined activities in the community that enhance resilience and belonging. Key program features involve the specialised Support Service Providers who assist DeafBlind community members with accessing and participating in essential services and social, vocational, educational, and recreational activities in the community. Support Service Providers were available for 80 service hours per month per DeafBlind individual and provided communication through American Sign Language, Protactile, and voice-over depending on which language was accessible for individual clients (Pei et al., 2019).

The evaluation was completed in partnership with the Connect Society, who commissioned a university-based evaluation team in Western Canada. The initial accountability focus of the evaluation was guided by three outcomes prescribed by the provincial government related to health, engagement, and independence. These outcomes evolved over time to more accurately reflect the work of the program’s responsivity to achieving clients’ person-centred goals. DeafBlind Support Services increased access to equitable services for DeafBlind adults experiencing isolation, dependence, and challenges with physical and mental health. Through the provision of a flexible, responsive roster of support service providers who could communicate in an accessible language, clients achieved visibility, voice, and agency. Specifically, the program:

• Improved physical and mental health for DeafBlind individuals with an emphasis on their personal fitness and well-being. This outcome was evident in the discussions with program stakeholders, who emphasised the importance of the program for their overall well-being.

• Increased inclusion opportunities for DeafBlind individuals as they engage in social interactions, community mobility, and envisioning new possibilities. This outcome was evident in the opportunities for connection and community.

• Enhanced independence and autonomy for DeafBlind individuals through provision of responsive services meeting their needs, embedding choice, and empowering them to overcome barriers.

• Created meaningful employment opportunities for Deaf and DeafBlind individuals, including Support Service Providers, Connect Society administrative personnel, and program participants. This proved to be an incentive for the funder to continue the program.

• Expanded program impacts within and beyond the DeafBlind community through continuing consultation, collaborative program development, and establishing networks for future scaling to new sites. This impact was unexpected and could be used in the future for increasing access to equitable service among low incidence, underserved communities that may be overlooked by existing services.

Methods guiding implementation of collaborative approaches to evaluation

The design of the study was reviewed and received approval from the institutional ethical review board. To describe our methods, we use the eight principles guiding collaborative approaches to evaluation (Cousins et al., 2020) and Whitmore et al.’s (2017) successful implementation criteria to build transformative partnerships.

1) Clarify motivation for collaboration

Open discussions with evaluation stakeholders are necessary to develop a shared understanding of the rationale and expectations for collaborative approaches to evaluation (Cousins et al., 2020). In our initial meetings, staff and clients emphasised to the evaluation team the pressing need for an evaluation to ensure that the program was able to continue to serve the DeafBlind community. The lead evaluators recognised that, for this particular evaluation, the unique cultural and linguistic context of DeafBlind Support Services would benefit from a participatory, collaborative, and transformative approach to evaluation and chose a collaborative approach to evaluation (Cousins et al., 2020; Mertens, 2017).

Stakeholders recognised the unique contributions of those involved in the DeafBlind Support Services evaluation activities to the co-generation of evaluation insights. The program manager provided necessary cultural and linguistic brokerage and firsthand lived experiences informing program needs. The Connect Society leader provided insight into the Deaf and DeafBlind culture and the circumstances in which the program had come to be. The graduate students on the evaluation team self-identified as novice evaluators, with one bringing experiences with the DeafBlind community and the funding body, the local provincial government. The faculty leads were both experienced program evaluators and researchers with established credibility in the community for working effectively alongside organisations. As the evaluation progressed, evidence of mutual learning and respect for the expertise of each evaluation stakeholder was apparent and identified as necessary for extending our collective capacity for an ethical, collaborative evaluation approach with a low incidence and underserved community.

2) Foster meaningful relationships

Sustained interactions among evaluation stakeholders are necessary to develop mutual trust that fosters meaningful relationships to support the collaborative approaches to evaluation processes (Cousins et al., 2020). Meaningful relationships among those involved in the evaluation were established and maintained through frequent conversations, via email and in-person meetings that included American Sign Language interpreters for the Deaf program manager. The evaluation team came to recognise the need to gather information using accessible signed, spoken, and tactile languages. As the evaluation progressed, the evaluators recognised the vital role that the program manager played as a linguistic and cultural broker. The manager, fluent in American Sign Language played roles as both a formal and informal connector of people involved in DeafBlind Support Services and the evaluation. Frequent, incidental, and planned conversations between program participants and staff achieved a fluent and responsive exchange of information that fostered relationships and advanced program outcomes. The lived experiences captured by the interviews made it possible to identify the broad range of outcomes and impacts. Relationships among evaluation stakeholders were important for mutual learning of the importance of the cultural and linguistic milieu of the program.

3) Develop a shared understanding of the program

Fostering a common understanding of the program is necessary in collaborative approaches to evaluation (Cousins et al., 2020). In the case of DeafBlind Support Services and its goals, processes, and history, in-depth discussions and document development with evaluation stakeholders were key contributors. Throughout collaborative approaches to evaluation implementation, evaluation stakeholders co-generated documents that were required to develop and confirm a shared understanding of the program, including a logic model. This collaborative work empowered those involved to make explicit how they conceptualised the connections between activities and program outputs.

An emergent need to develop a policy brief provided an unexpected opportunity to develop a document for funders. The policy brief aligned evaluation findings with the government’s policies to support DeafBlind Support Services staff’s quest to secure ongoing funding. DeafBlind individuals and their family members, representatives of a DeafBlind advocacy organisation, service providers, interpreters, a business leader, and program administrators led an open discussion with the government officials. Evaluation team members were available to respond to inquiries about the policy brief and evaluation outcomes. The presentation prompted a shared understanding of the emergent outcomes and impacts of the program among evaluation stakeholders. The development and feedback process inherent to collaborative approaches to evaluation was essential for developing a deep understanding of the nuances of DeafBlind Support Services to communicate shared understandings of the program among evaluation stakeholders and funders.

4) Promote appropriate participatory processes

Culturally and linguistically appropriate evaluation procedures, constructs, and communications were embedded in an ethical methodology identified as necessary for genuine participatory engagement (Rodrieguez & Acree, 2021). To encourage the participation of evaluation stakeholders, comfortable and accessible environments for data collection were important (Cousins et al., 2020). Distinct perspectives, experiences, and expertise among evaluation stakeholders guided decisions about evaluation plans and activities. Accessible communication modes for Deaf and DeafBlind individuals, including English text, American Sign Language, Protactile, and a video relay service, were used to facilitate lived experience interviews and evaluation meetings. Clients’ participation in evaluation activities was voluntary, as was made clear during the informed consent processes.

The interviewer took time to ensure clients understood the ethical parameters including informed consent and confidentiality. Appropriate participatory processes were further promoted by involving the Connect Society leader and DeafBlind Support Services program manager throughout the evaluation. This included identifying data sources, co-creating interview questions, reviewing and offering feedback, and participating in the writing of this article for publication. By providing accessible language, Deaf and DeafBlind program staff and clients were empowered to participate fully in evaluation processes. These steps facilitated appropriate participatory processes within a culturally and linguistically unique context.

5) Monitor and respond to the resource availability

Paying attention and responding to resource constraints is a critical component of collaborative approaches to evaluation (Cousins et al., 2020). The ethical imperative for evaluators is to find ways to gather meaningful data within the boundaries of resources available. Initial conversations revealed several opportunities that could be leveraged at no cost. DeafBlind Support Services administrative program data could be incorporated into the data collection, analysis, and report writing. Report writing could be embedded into the coursework of one of the graduate students. Through creative thinking and careful planning, evaluators identified key areas of expertise and leveraged opportunities that could mitigate resource constraints.

6) Monitor evaluation progress and quality

Frequent conversations to monitor evaluation progress and ensure quality outcomes is key to collaborative approaches to evaluation (Cousins et al., 2020). During the evaluation, ongoing and iterative dialogues among evaluation stakeholders fostered understandings of existing data and how quality data could be collected to ensure accuracy of findings. Evaluators maintained data quality and professional evaluation standards using collaboration to promote quality data collection. For example, the initial coding was completed by one graduate student, and another provided interrater reliability checks. A review by evaluation team leads followed the initial drafting of the evaluation report and findings by graduate students. Once a fulsome draft was available, the report was then discussed among all the evaluation stakeholders and feedback was provided. From this discussion, changes were made, and consensus was reached that the findings were credible and in keeping with the available data and the experiences of program participants. The integrity of the evaluation’s processes and findings were monitored through collaborative processes and the participation of evaluation stakeholders.

7) Promote evaluative thinking

Discussions about emerging understandings with stakeholders provided opportunities to learn about and engage in evaluative thinking (Cousins et al., 2020). Evaluative thinking capacity was built among stakeholders with a focus on learning and inquiry. Ongoing discussions about the evaluation goals, data, and emerging understandings served to create a space to unpack ideas through systematic inquiry among evaluation stakeholders. At each opportunity, evaluators exchanged ideas and sought diverse perspectives, intentionally pursuing mutual understanding. This guided the interpretation of the findings and stimulated further discussions. For example, preliminary evaluation results were explored through the co-generation of a policy brief. The policy brief presented an early opportunity for the evaluation team to share perspectives and understandings of preliminary findings. This process was then continued in the co-development of the evaluation report. Providing multiple opportunities to share perspectives through the co-creation of documents was essential for promoting evaluative thinking among evaluation stakeholders.

8) Follow through to realise use

Involving stakeholders in actioning evaluation outcomes is key to collaborative approaches to evaluation (Cousins et al., 2020). The dissemination of meaningful evaluation outcomes helped to shift how organisations and individuals viewed the program. Through embedded evaluation and collaboration, evaluators were able to highlight for staff, funders, and advocates the evaluation findings that informed the sustained implementation of program activities. Evaluators and program staff were receptive to the identification of unanticipated outcomes, specifically the employment of Deaf and DeafBlind individuals and the potential for scaling. DeafBlind Support Services and its parent organisation, Connect Society, provide employment for individuals who have typically found it challenging to access meaningful work. Specifically, DeafBlind Support Services provides employment opportunities for Deaf Support Service Providers, a Deaf program manager, a DeafBlind booking agent, and Deaf and DeafBlind guest speakers and trainers. DeafBlind individuals attend the same educational workshops as Support Service Providers, which can increase their capacity to find meaningful work. The initial constructs of health, engagement, and independence remained salient outcomes for clients, and employment was viewed as another pathway to well-being.

Continued, sustainable financial support for Connect Society to maintain and expand DeafBlind Support Services was secured and its role was affirmed as an effective and responsive not-for-profit community service provider. The collaborative approach to evaluation provided new insights into evaluation, program impacts, and the program’s future potential. This includes a potential for extending DeafBlind Support Services implementation and the evaluation work to other populations requiring support to live independently and fully in their community. Ongoing collaboration, embedded within program goals and outcomes, were identified as important for scaling, adaptability, and monitoring the impacts and implications of DeafBlind Support Services.

Implications for ethical collaborative approach to evaluation practice

In describing our co-generated evaluation insights, we apply recent work of Mertens and Catsambas (2021) and Chouinard and Cousins (2021) to advance an ethical imperative describing philosophical foundations, pragmatic actions, and political leadership for implementation of a collaborative approach to evaluation with a low incidence and underserved community. Mertens and Catsambas (2021) describe examples of applying a transformative lens to program evaluation. With this lens, the program evaluators were able to recognise the social innovation inherent in a support service program led and informed by members of a low incidence community. Chouinard and Cousins (2021) consider the ethical implications of foundational philosophical, pragmatic, and political justifications to describe the ethics of engagement among evaluation participants. The ethics of engagement include reflexivity, relationality, responsibility, recognition, representation, reciprocity, and consideration of rights, which intersected with the eight principles of collaborative approaches to evaluation and a transformative program evaluation lens (Chouinard & Cousins, 2021; Cousins et al., 2020; Mertens & Catsambas, 2021). Our reflection on this evaluation experience offers three ethical implications for collaborative approaches to evaluation implementation with a low incidence and underserved community.

Philosophical foundations: Equitable partnering and learning

Equitable partnering by all those involved in the evaluation identified shared goals through mutual learning. Underlying our equitable partnerships were philosophical considerations for the unique experiences and perspectives of all evaluation stakeholders. Connect Society recognised the need for partnership with a credible evaluation team to meet funding requirements. The evaluation team were committed to understanding unique the contexts and complexities facing the DeafBlind community. The use of data collection and analysis practices honouring the cultural and linguistic uniqueness of the Deaf and DeafBlind communities contributed to building mutual trust. In so doing, collaborative approaches to evaluation provides a foundation for achieving shared goals and mutual trust through equitable partnering and learning with low incidence and underserved communities.

Pragmatic actions: Capacity building and creativity

The commitment of the program staff to build their capacity to deliver impactful service with their community and creativity on the part of the evaluation team to leverage limited available resources were important considerations for the success of the collaborative approach. The evaluation team streamlined the evaluation design to support evaluation feasibility and meet funder timeline requirements. Practical aspects of the evaluation process and products included drawing upon program resources, such as existing administrative program data and using Connect Society roster of interpreters to translate interviews with DeafBlind program clients and the program manager. Program staff engaged with the process and built capacity to consider the value of the data sources available for program evaluation. With a responsive and creative approach to working with the resources available, both the program and the evaluation team built capacity in new areas of evaluation expertise and collectively developed new insights about the dynamic influences of the program, its context, and population. In so doing, collaborative approaches to evaluation with a low incidence and underserved community offer implementation guidance that creates conditions for stakeholder capacity building and creativity in evaluation activities that fit with culturally and linguistically unique contexts.

Political leadership: Sustained participation and responsiveness

Participation by Deaf and DeafBlind individuals as evaluation stakeholders and responsiveness to emerging opportunities to elevate community member voices and experiences were important considerations throughout the collaborative approaches to evaluation process. Sustained participation by the low incidence and underserved community members to share the life-changing impacts of DeafBlind Support Services at a critical time when funding was at risk was essential to the success of the evaluation to secure ongoing funding. Responsiveness by the evaluation team to create conditions for meaningful stakeholder participation was inherent to the evaluation process. In so doing, collaborative approaches to evaluation provide a framework for evaluation for empowered participation and responsiveness that can influence political decisions to the benefit of low incidence and underserved communities.

Conclusion

This practice article describes our experiences of implementing collaborative approaches to evaluation with a low incidence, underserved community under severe resource constraints to advance an ethical imperative for those using or planning to use collaborative approaches to evaluation in two important ways. First, we action our ethical rationale for collaborative approaches to evaluation through co-generating our evaluation insights and second, we build transformative partnerships through the application of the eight principles guiding collaborative approaches to evaluation. A respectful, inclusive, and participatory approach to collaborative approaches to evaluation was achieved with the application of the eight principles. Through ongoing conversations among the evaluands, evaluators, and program participants and the use of available data sources, the evaluands were able to articulate the benefits and funding was renewed for the program. Finally, we advance an ethical imperative for collaborative approaches to evaluation by describing the philosophical foundations, pragmatic actions, and political leadership for its implementation with a low incidence and underserved community. It is our hope that others can learn from our experiences and to further extend the application of collaborative approaches to evaluation in complex contexts.