Abstract
Ripple effects mapping (REM), a qualitative participatory approach to intervention evaluation, is gaining recognition as a useful method for elucidating the long-term intended and unintended impacts of complex public health interventions. The present study applied an adapted REM approach to capture systems and community change associated with the Agenda Gap program. This population-level youth mental health promotion intervention is embedded in multi-sectoral partnerships with long-term and relational outcomes post-program that are difficult to elucidate using traditional program evaluation methods. Using transcript and mind map data generated through an REM process with former Agenda Gap youth collaborators and adult allies, reflexive thematic analysis supported the construction of three thematic program outcomes: (1) Reimagining Future Possibilities, (2) Systems Integration: Transforming School Practices, and (3) Progressing From Ripple Effects to Sea Change. Spanning socioecological levels (i.e., individual, family, community, and societal), the outcomes and their associated sub-themes captured the meaningful impacts experienced by Agenda Gap participants, as well as those more distal to the intervention, in the years following implementation. These findings demonstrate the substantive, multi-level impacts of the program and also illustrate how qualitative, participatory approaches, such as REM, can complement other forms of evaluation to reveal outcomes that are typically overlooked. Recommendations and implications for future research and applications of REM are offered.
Keywords
Introduction
Youth mental health is a priority public health issue globally (McGorry et al., 2024). Indeed, the last several years have brought a new sense of urgency to respond with interventions spanning both clinical and community contexts to improve outcomes for individuals and for broader populations; in other words, population-level, system-facing outcomes beyond individual health or behavior change (Benton et al., 2021). However, while approaches to measuring the effects of such efforts are well established for clinical initiatives or those targeting change in individual-level outcomes and over the short term, the study of broader public health interventions and their longer-term and equity impacts is less well established (Rutter et al., 2017). Ripple effects mapping (REM) is a qualitative method increasingly recognized within the research community as a useful approach to assessing the effects of complex (i.e., multi-level and multi-component) public health interventions (Nobles et al., 2022). Yet, more research is required to fully understand the utility of REM for elucidating the impact pathways and outcomes of such initiatives. To address this evidence gap, the present study incorporated REM into the evaluation of Agenda Gap, a complex public health intervention advancing youth mental health through policy engagement (Jenkins et al., 2023).
Agenda Gap: Intervention Overview
Developed in partnership with youth from diverse backgrounds in British Columbia, Canada, the Agenda Gap intervention is grounded in the core tenets of mental health promotion, positive youth development, community youth development, and liberation psychology (Jenkins et al., 2020; Jenkins et al., 2023). In this way, it is designed to bolster the mental health and well-being of youth and their communities using a human rights approach to action on the social determinants of health and equity. With youth expertise at the center, youth collaborators (aged 12–24 years) recruited through partner organization networks (e.g., schools, community organizations, and health services) are empowered with the skills and knowledge to advocate for policies that strengthen mental health across all socioecological levels—for individuals, families, communities, and society. Over a six-month period, cohorts of 5–15 youth with shared experiences or interests engage in weekly two-hour facilitated sessions. During this time, they explore interconnected concepts related to mental health promotion, social and structural determinants of mental health and (in)equity, youth rights as a platform for policy advocacy, and systems change. Adult allies, who serve in policy and other decision-making roles, support the youths’ system change efforts. They are provided with materials and guidance to help them facilitate meaningful youth engagement in policymaking during the intervention and following its conclusion. Additionally, multi-pronged knowledge mobilization strategies are employed to extend the impacts beyond direct program participants (see Jenkins et al., 2020, for a complete description of the Agenda Gap intervention content and protocols).
To date, Agenda Gap has supported a total of 167 youth across 15 groups located in British Columbia, Alberta, Ontario, and Nova Scotia. Intervention participation has prepared these youth to surface policy priorities and actively engage in policy processes focused on mental health and equity across all socioecological levels. Partnerships with local community organizations, school districts, youth clinics, and policy experts have expanded the program’s youth-driven impacts, leading to long-term, sustained outcomes that are challenging to capture. Previous evaluations of the program, employing mixed and realist evaluation methods, have identified temporally related results anchored in the intervention theory and components (Jagosh et al., 2024; Jenkins et al., 2023). However, Agenda Gap features a complex, multi-component design and dynamic implementation process, and the potential for multi-level impacts. Thus, assessing “what works” is complicated, and evaluation necessitates a novel approach that can accurately capture the full scope of intervention impacts and outcomes across all socioecological levels and over time.
Ripple Effects Mapping: Method Description
REM is a participatory, qualitative evaluation method used to learn from participants and others with direct or indirect knowledge of an intervention about the ways in which an intervention has contributed to various outcomes or systems-level changes (Chazdon et al., 2017; Nobles et al., 2022). Originally developed to support impact evaluation of community-based programs, REM entails four core elements: (1) appreciative inquiry, (2) a participatory approach, (3) interactive group interviews and reflection activities, and (4) radiant thinking through, for example, mind mapping (Ali et al., 2021; Chazdon et al., 2017). In a typical REM data collection event, participants share stories through structured one-on-one interviews guided by appreciative inquiry questions to surface key successes, new or deepened relationships, and any “ripples” that have resulted from intervention participation or implementation. Subsequent share-back in a group setting is used to prompt broader reflection and to explore program effects with a facilitator visually mapping the resulting ripples—or the chain of events associated with intervention impacts and outcomes—to produce a ripple effects mind map. While there are different ways to approach REM mind mapping, a central feature is the visual depiction of participant-identified outcomes and associated events. Mind mapping not only supports the co-creation of data but also stimulates deeper reflection and dialogue (Umoquit et al., 2013; Wheeldon, 2011), which helps to ensure that a collective, and comprehensive understanding of intervention impact is constructed.
The four core elements of REM are foundational across the various REM approaches that have been detailed (Chazdon et al., 2017), with differences centering chiefly on the mapping scope, approach to coding (i.e., deductive vs inductive), and whether analysis and interpretation is done during or following the REM event. REM has been applied to the evaluation of a range of community-based interventions, including those focused on childhood obesity prevention (Olfert et al., 2019), health volunteer capacity building (Washburn et al., 2020), youth development (Emery, 2013), and youth mental health promotion (Spain et al., 2021), wherein it has demonstrated its utility in elucidating both intended and unintended intervention impacts.
Ripple Effects Mapping and Complex Public Health Intervention Evaluation
Recently, researchers have begun to adapt REM as a tool to assess complex public health interventions. At the forefront of this work, Nobles and colleagues (2022) leveraged REM to evaluate a systems approach to physical activity implemented in the United Kingdom. Their application featured an adaptation of REM methods to expand its use both during and following intervention implementation and incorporated a timeline into the mapping process. In addition to developing themes to reflect pathways of intervention impact, the researchers also combined their REM findings with other evaluation results to achieve a holistic understanding of the program. Collaborating with other researchers, they then worked to produce recommendations for strengthening the REM approach, including employing strategies to ensure REM is reciprocally beneficial for all involved and considering the effects of power imbalances on REM engagement and outputs (Creaser et al., 2024). Additionally, these researchers have also advanced a realist-informed REM approach to move beyond impact cataloguing to delineate the causal mechanisms of impact pathways—a process previous REM approaches were not theoretically or philosophically designed to support (Harris et al., 2024). This more intensive process requires detailed knowledge, and often more time from participants, to answer questions about the “why” and “how” of systems change efforts (Harris, 2018). These recent applications of REM to complex population-level interventions suggest its potential for capturing impact pathways and intended and unintended outcomes across socioecological levels and in the years following intervention delivery.
The Present Study
Given some of the limitations of traditional evaluation methods, the present study applied an adapted REM approach (described in full below) to capture the breadth of long-term multi-level impacts and outcomes of the Agenda Gap program. Using REM adds to the burgeoning area of complex public health intervention research and responds to calls for researchers to include evaluative follow-ups to more accurately reflect the effects and duration of intervention outcomes and to contribute to “clearer and more convincing advocacy messages to influence the support for prevention and promotion interventions” (Jané-Llopis & Barry, 2005, p. 52).
Methods
REM Data Collection
Participants for the present study were recruited from one of the first cohorts of the Agenda Gap program, which was implemented in 2020–2021 with a group from a suburban community located in British Columbia, Canada. Youth participants (n = 12) and adult allies affiliated with the school district (n = 5), non-profit community organizations (n = 3), and health services delivery (n = 2) were invited through email to participate. Ethical approval for the study was obtained by The University of British Columbia’s Behavioural Research Ethics Board (H17-001602). Informed consent was obtained prior to the REM process.
Procedure
REM Approach and Adaptations
We implemented an adapted REM approach inspired by Chazdon and colleagues (2017) using an online format (via Zoom) with synchronous mind mapping (via Miro) and asynchronous (i.e., offline) analysis and interpretation. Given our aim to capture the breadth of both intended and unintended intervention impacts (i.e., those that align with our initial intervention theory and those that extend beyond) (see Jenkins et al., 2020, for an overview of the Agenda Gap intervention theory), we also adopted a ripple mapping process more typical of what Chazdon and colleagues (2017) conceptualize as “theming and rippling” with “in-depth rippling” to support analysis. Modifying the rippling approach reduced the time required for the REM event by moving analysis and interpretation offline, thus reducing the burden of participation while still enabling participant voices to be centered.
The two-hour session took place in June 2024, approximately three years post-intervention (2020–2021). We purposefully recruited participants with varying levels of knowledge about the program and its impacts, including youth alumnae and adult allies, as well as one ally not directly involved in intervention delivery, who could together provide evidence for both proximal and distal ripples across system levels. The session was attended by three members of the research team to support the process. To minimize bias and potential power imbalances, one researcher, who had not been involved in intervention delivery but who had previously conducted post-intervention qualitative interviews with Agenda Gap participants, facilitated the event. The other two researchers, one of whom had been involved in the implementation of the Agenda Gap intervention, made observational notes, populated the event mind map using Miro, a digital mind-mapping software, and offered follow-up questions.
Appreciative Inquiry Questions for REM Partner Interviews
As stories were shared, the facilitator invited other participants to contribute their ideas and reflections and asked probing questions to elicit details, such as who else was involved, when particular events occurred, what else was learned, and how that information was used. This supported the identification of additional ripples, their chained connections or “ripple pathways” (who and what was involved) over time, the specifics of the socioecological levels involved, and unintended impacts associated with the program or its ripple effects. Over a period of approximately 60 minutes, the research team recorded key ideas and concepts as they emerged using the on-screen mind-mapping tool. See Figure 1 for the mind map developed during the session. Ripple Effects Mind Map Created During Online REM Session
Following the sharing of stories, the facilitator led a group reflection on the overall REM process, asking participants to share what they found most interesting during the discussion, new learnings, or recommendations for mobilizing findings to extend their impact. The entire session was audio recorded via Zoom to produce a transcript. Following the conclusion of the REM event, the transcript was verified against the recording, and the mind map was then examined alongside session notes and the transcript to further organize and clarify the map content. Participants were then invited by email to review the cleaned transcript and map to ensure it represented their interpretation of the impacts and outcomes shared during the REM session (see Figure 2 for a visual depiction of the adapted REM process and components utilized). Adapted Online REM Process
Data Analysis
We conducted analysis of the data generated using reflexive thematic analysis (Braun & Clarke, 2006, 2019). Following the REM session, we collaboratively progressed the analysis from a descriptive (i.e., organized into patterns to show content) to a more interpretive level of analysis (i.e., where underlying conceptualizations shaping the content were identified). This process involved treating the mind map as the primary data artifact and the audio transcript as supporting context. We exported map nodes (i.e., concepts, ideas, and topics) and edges (i.e., connections between related concepts, ideas, and topics) and coded at the level of ripple pathways, annotating each with the socioecological level of outcome/impact (i.e., individual, family/interpersonal, community, and societal), time horizon (i.e., short term ≤ 12 months, medium term = 12–24 months, and longer term ≥ 24 months), and valence (i.e., unintended vs. intended). Ripple pathways were then grouped inductively into outcome areas, after which we produced a revised ripple map (see Figure 3) and a pathway table to make the connections—both intended and unintended—explicit. Grounding theme development in the transcript data privileged participants’ viewpoints and communicated impacts in a manner that may be useful to their own knowledge mobilization goals. It also provided transparency to participants about how themes were constructed and the socioecological levels assigned, supporting their ability to replicate our decision-making and provide critique. The revised map, themes, and table were shared with REM participants and validated via email. Given their interest and extended engagement supporting intervention delivery, and as a measure of reciprocity, one of the participants, an adult ally, was invited to contribute to the analysis and co-author this publication. Ripple Effects Mind Map
Results
A total of six people, two former Agenda Gap youth collaborators and four adult allies, participated in the REM event. Youth participants were girls/women from non-white ethno-racial backgrounds, aged 20 and 21 years. The four adult allies identified as women (n = 2) and men (n = 2) and worked in the education (n = 2), community development (n = 1), and health services (n = 1) sectors. All but one of these allies had directly supported the Agenda Gap youth to varying degrees during the intervention delivery. The fourth adult ally had become aware of Agenda Gap through the intervention’s longer-term, community-level impacts.
Thematic Outcomes and Sub-Themes Organized by Socioecological Levels
Reimagining Future Possibilities
A prominent intervention outcome within the REM data was increased capacity among individuals and groups to reimagine future possibilities. Within this outcome area, participants’ stories and reflections centered on two sub-themes: (1) new understandings and modeling of a relational approach and (2) expanded youth leadership. The mind map visually depicted this outcome area in the upper right quadrant under the cluster labeled “YOUTH REFRAMING POSSIBILITIES.” The visual structure showed a temporal progression, with youth’s engagement in decision-making and shifted focus of mental health initiatives to include upstream emphasis (represented as a medium-term box) connecting through multiple pathways to longer-term outcomes (triangles) including career aspirations and expanded youth leadership capacity.
Regarding the first sub-theme, participation in Agenda Gap provided an opportunity for youth to be exposed to new ways of thinking about and acting to improve mental health, while also exposing them to a relational facilitation style that encouraged a sense of safety and growth. Youth described this process as enhancing their self-confidence as they integrated these approaches into the creation of student-led school-based mental health clubs, as well as into their daily lives. As one youth participant shared, these learnings supported them to “not only become more comfortable expressing [themselves] but to hold the space to be able to understand others as well” (Participant 1), which guides their current work in youth suicide prevention. This further strengthened their sense of confidence and grew their expertise “personally and professionally.”
Youth also shared that their participation in Agenda Gap and related advocacy for intergenerational dialogue programming on the topic of mental health influenced their educational and career interests and goals, moving them outside of cultural norms and expectations. Participant 1 explained that being involved in Agenda Gap “definitely helped solidify my career path in psychology,” especially as someone “growing up in a culture that doesn’t really go into mental health, it doesn’t consider having a career in it.” Adding to this, Participant 3, an adult ally, reflected: Agenda Gap planted seeds for students to make this their life’s work and to take what they’ve learned and grow this, [which will contribute to] the mental health landscape and building capacity to bring people into this field who can do this work and continue to grow the kinds of changes that need to happen so that everyone feels supported, everyone feels heard, everyone feels connected.
Youth and adult participants agreed that the Agenda Gap intervention provided an opportunity for youth and adult allies to become more passionate about youth well-being. This progression was visually captured in the mind map through arrows connecting “Youth integrate facilitation approaches into their own initiatives” with “Facilitation approach modeled for others,” demonstrating the mutual influence between learning and leading. They provided further evidence of this as they reflected on federal government funding that was awarded to a more recent cohort of Agenda Gap youth to support their efforts related to school-based mental health needs. This funding created opportunities for youth to explore and respond to gaps within school settings, which the REM participants described as “impactful and inspiring,” leading them to reflect, “what we say is important, too, and what we can do can make an impact as well” (Participant 1).
Systems Integration: Transforming School Practices
Another thematic outcome constructed from the data involved systems integration, resulting in the transformation of school-level practices. Participants’ stories highlighted two sub-themes: (1) centering youth voice and mental health and (2) prioritizing family engagement in conversations about youth mental health. The density and interconnectedness of these school-level transformations were visually apparent in the mind map’s center-left region, where the “APPRECIATION OF YOUTH VOICE” cluster and “YOUTH-DRIVEN CHANGEMAKING” cluster contained numerous medium-term (boxes) and longer-term (triangles) impacts radiating outward. The establishment of a district youth advisory committee that employed Agenda Gap principles appeared as a box, indicating its medium-term emergence, with connecting lines showing its influence on subsequent policy changes.
Regarding the first sub-theme, participants described a noticeable shift within their school district related to the importance of youth voice and mental health. This was evidenced by a new emphasis on youth-engaged decision-making, which they connected, in part, to the Agenda gap intervention. Further, they shared that intentional relationships continue to be built to support and prioritize youth well-being. For example, since the 2020–2021 implementation of Agenda Gap in this community, the school district has brought in new mental health literacy programming for students in grades 6 through 8, with former Agenda Gap youth and adult allies actively supporting these initiatives.
Following the culmination of this Agenda Gap cohort, a commitment to youth voice and mental health has continued in the school district. For example, a district youth advisory was created that employed some of the Agenda Gap principles and involved the youth collaborators. For Participant 3, the work spearheaded by Agenda Gap participants contributed to a systems-level transformation in which schools now authentically listen to students: There’s a better understanding of what youth voice actually is—student voice … we’re understanding that we have to listen in a way that allows for action, right? That’s a change. That it isn’t just, “OK, we heard you. That’s enough. Stop.” And I think that’s been one of the biggest changes that I’ve seen for sure in terms of authentic student voice in our district.
As a result of this emphasis on student voice, this participant and other school officials are working to ensure that their updated strategic plan incorporates goals related to student mental health and well-being. This strategic planning shift was represented in the mind map as “Mental health increasingly prioritized in school district strategic planning” (a longer-term triangle), with multiple connecting lines showing how earlier impacts, such as youth-led initiatives and the district advisory committee, contributed to this systems-level change. This participant further expressed: It’s very satisfying to see all of those pieces and how much that work is informed by our student voice and … to actually see that now become concrete … You know, the current [strategic plan] does not have mental health—it’s there, but it’s implied and kind of embedded. We want to make it more clear and sort of front and center. And so I think that does come from Agenda Gap because that’s the purpose of the work that we were doing and our students were talking about … So in this work, doing the framework, we’re hoping to actually make it part of the conversation of the new strategic plan that’s coming together next year. (Participant 3)
These efforts to center youth voice in decision-making and actively prioritize youth mental health in strategic planning provide further evidence that the participants perceive Agenda Gap as affecting change across the system and becoming firmly integrated into the community at multiple levels.
One participant described how mental health data, including that which capture student perspectives, are now available and being used to further strengthen capacity for school leadership responses. This commitment was evident to youth participants, with one reflecting: I remember when we did the dialogue, like, four years ago. I remember being so touched and honestly surprised by how involved so many of the administrators were … where I was, like, adults really care. And now … it feels like it’s becoming a norm that adults are going to be around, they’re going to be supporting you. And it doesn’t feel like the shock anymore, or something that you have to beg for. So, seeing that it’s becoming a norm and it’s so much more organic for youth to be connected to administrators, and their voices to be more prioritized, is incredible to see. (Participant 1)
These shifts in adult practices, where leaders actively seek youth input and engage with them and their priorities authentically, suggest personal growth among the adult champions of Agenda Gap, and indeed growth in leadership capacity more broadly. The mind map captured this transformation through participant language embedded directly in the visual: “Listen in a way that allows for action” and “Feels different—you could feel the youth leadership,” providing both data points and evidence of the cultural shift. Overall, the expansion of programming that taps into and amplifies youth voices in schools, along with wider shifts in community norms, practices, and culture, helps to substantiate participants’ perceptions of the positive impacts of Agenda Gap within and across system levels.
Related to the second sub-theme, participants shared that the policy advocacy led by this cohort of Agenda Gap youth subsequently brought together over 200 parents, which “empowered [the students] to see that their perspectives mattered” (Participant 3). Families were invited into intergenerational conversation with youth, where they were able to hear “directly from the students and then [break] off” for small group sessions in both English and Mandarin. Through parent feedback, the school community heard “a lot of humility in terms of parents acknowledging that they are learning and … not knowing a lot … there’s a thirst from parents to learn more about mental health literacy and how they can best support their kids” (Participant 5). REM participants further elaborated on observing a new commitment from the education sector to engage families on the topic of youth mental health. This change was attributed to the youth advocacy resulting from the Agenda Gap intervention. This has led to family learning events, as well as best practices to involve parents and caregivers through culturally relevant and responsive methods. As one of the adult participants shared: When you’re coming from a home setting where discussions around mental health aren’t always at the top of the agenda and you’re dealing with all sorts of stigma or reluctancy to engage in certain topics, having some of those additional supports in our school district can benefit the youth and their families as well. (Participant 5)
Clearly, REM participants perceived great value in prioritizing family engagement in conversations about youth mental health and viewed family learning events as key in shifting perceptions about mental health, in particular working to remove stigma through open discussion and acknowledgement of mental health challenges, thereby opening the door to meaningful and impactful dialogue and change.
Notably, evidence of decreased stigma related to mental health was identified by participants as reaching beyond families. A community-based adult participant observed that best practices in family engagement were also leading to destigmatizing practices in the broader school community: I do think that this school has actually been doing a really good job over the last three or four years. Walking the talk, so to speak. I’ve been noticing more [Parent Advisory Councils] reaching out for presentations. I’ve been noticing more purposeful interactions and connections with the school district … I do feel like it has been more proactive and active than reactive in the last few years … it just takes a bit, a few ripples to kind of make a wave. (Participant 6)
Evidence of the lasting long-term impacts of Agenda Gap was apparent across youth and adult participants’ stories, which highlighted the ways that program outcomes continue to “ripple” through the education system in support of youth mental health and well-being. This metaphor of ripples becoming waves, offered by Participant 6, was visually apparent in the mind map structure itself, with the majority of family engagement outcomes appearing as medium- to longer-term markers, demonstrating the sustained nature of these changes.
Progressing From Ripple Effects to Sea Change
Beyond systems-level changes within school settings, participants also shared outcomes extending to the community and societal levels through policy innovation, which was captured within the sub-theme (1) youth voice informing policy and practice. The long-term ripple effects of the Agenda Gap intervention helped to initiate and embed mental health learnings in new spaces throughout the community. This included guiding approaches to youth advisory roles in health services settings and shifting practices in support of youth mental health. The mind map contained multiple longer-term outcome markers (triangles), visually demonstrating how ripples progressed to broader systems change. The city’s “10-year youth strategy,” one of the most distal outcomes captured, had connecting pathways tracing back through medium-term impacts such as youth serving as advocates in health services and changes in language and practice. Dashed lines connecting to several policy changes revealed unintended pathways of influence that participants had not initially anticipated when the intervention was designed.
Following the implementation of Agenda Gap in this community, at least three youth were invited to join an advisory group to inform the practices within a youth health services organization operating through the local health authority. One participant explained that in this role: [The youth] really solidified the terms of reference, the roles, and more, the kind of mission and vision of the group itself. And it still continues to this day. None of the three Agenda Gap youth are part of that anymore. I think they moved onto bigger and better things, but ... there are a lot of their fingerprints on, like, the initial documents that they still use. (Participant 6)
This participant also shared that the evolution around the valuing of youth voice and expertise has extended to the municipal government, including informing language in the city’s youth strategy and creating opportunities for meaningful youth engagement. For example, they described that youth are now included in government policymaking through the development of a youth advisory committee: The city just launched a youth specific advisory committee. So youth picked from [this community] will have a voice in speaking to council or a voice and feedback in policy and procedure for the city itself ... (Participant 6)
Another participant confirmed that they saw Agenda Gap principles along with the youth’s advocacy language reflected in the strategy once released. The term “intergenerational dialogue” showed up in that [committee] document, and I immediately thought, ohh, there’s Agenda Gap, because that was language that the group developed and we still use to this day in the district. (Participant 3)
Participants recognized program influences across their community to reach and impact beyond direct Agenda Gap participants and allies. The mind map visually represented this extended reach through a pathway connecting “AGENDA GAP PROGRAMMING” (the central hub) through multiple intermediary nodes to societal-level outcomes such as “Increased youth participation in civic and government policy” and “Change in government policy and procedures,” with the majority of these connections marked as longer-term (triangles), indicating their sustained emergence over the three-year period. Further, participants recounted that youth voice continues to be prioritized as policies and practices evolve. Having long-term champions of these issues in community was considered key in transforming ripple effects into broader and longer-term changes and equity impacts. Near the close of the REM session, one of the adult participants shared their perspective as an elected school system leader who has had the opportunity to observe and experience program effects from the “outside”: This is so rich. I’m hearing ripple after ripple after ripple … You can feel it and just thank you for letting me be a part of this because I’m sitting here thinking, Ohh, my gosh … We need to have Agenda Gap be a part of presenting to the rest of our board … And I feel, again, that feeling … it’s changing … (Participant 4)
These sentiments resonated with all of the participants—how the ripples had magnified into what one referred to as a “wave,” leading to a sea change they aimed to continue to support and sustain. The visual map captured this progression powerfully, with Participant 4’s observation of “ripple after ripple after ripple” mirrored in the cascading pathways extending from the program core to the map’s periphery. The concentration of triangular markers in the outer regions provided visual evidence of the cumulative, long-term nature of systems change, supporting the theme of progression from ripple effects to a sea change.
Discussion
The present study applied an adapted online version of REM to evaluate the Agenda Gap, a complex public health intervention focused on strengthening mental health of youth and their wider communities. This supported new insights into the broad and sustained impacts of the program, while foregrounding the perspectives of intervention participants as well as those more distal to the program. These impacts were both intended and unintended, as predicted by the initial intervention theory, and revealed evidence of shifting norms and practices leading to outcomes that continue to ripple through the community three years post-intervention. The REM approach enhanced our ability to elicit more fulsome and precise accounts of intervention outcomes and their duration, providing compelling evidence for investment into promotive and preventative youth mental health interventions to more comprehensively respond to this leading public health challenge. These findings also add to the growing research base demonstrating the utility of REM to evaluate complex, multi-component, and multi-level public health interventions (e.g., Creaser et al., 2024; Nobles et al., 2022) and suggest that including this approach as part of a broader program of evaluation can help discern the less visible long-term intended and unintended program impacts that operate to produce meaningful outcomes across socioecological levels.
Intervention Impacts
The use of REM enabled insights into the impacts of the Agenda Gap intervention beyond those identified in previous research. Indeed, our team has used a variety of methods to evaluate Agenda Gap, including mixed-methods and realist approaches, but gaps remained in terms of capturing the mid- to long-term impacts and outcomes across socioecological levels. For example, analysis of quantitative data from an earlier mixed-methods evaluation of Agenda Gap (Jenkins et al., 2023) identified proximal outcomes among youth collaborators, including gains in knowledge about mental health concepts (e.g., resilience, mental health, youth rights, and intersecting vulnerabilities) and improvements in positive mental health constructs (e.g., self-efficacy, resilience, peer support, and adult support). Data from post-intervention individual qualitative interviews with youth collaborators and adult allies evidenced themes of impact at the individual, family, and community levels, which together pointed to promising shifts in participants’ understandings of mental health and the potential for the program’s socioecological approach to mental health promotion resulting in real and lasting effects. Yet, that these data were collected shortly following the intervention’s completion makes these implications more tenuous. Incorporating REM into our program of evaluation provided rich learnings about intervention outcomes from the perspectives of Agenda Gap participants as well as others who were more distal to the process but who had observed program benefits in their community. These insights and reflections offered by the REM participants and constructed into themes add another important metric to help define and monitor the intervention’s multi-level and sustained effects. Accurately demonstrating the impacts of complex, multi-level, community-based interventions is a challenge recognized by other public health scholars (Blankenship et al., 2006). Based on our findings and others’ (e.g., Creaser et al., 2024; Harris et al., 2024; Nobles et al., 2022; Spain et al., 2021), the REM method presents a promising participatory tool to augment and support the comprehensive evaluation of intervention impacts, both short- and long-term, yielding insights that would not have been possible using a single or strictly quantitative methodological approach.
Core elements of the REM process enabled deeper insights into Agenda Gap’s lasting, yet previously obscured impacts. For example, mapping of the session content helped participants with outcome recall, and growing and connecting stories, while increasing recollection and depth (Meyer et al., 2013). Indeed, this participatory approach is helpful in capturing what is important to participants following the implementation of an intervention, inclusive of less observable shifts (i.e., mindsets and infrastructure) (Nobles et al., 2022). The REM process, featuring shared discussion and an opportunity to build on each other’s ideas, allowed participants to see contributions extending from program implementation from multiple perspectives, such as shifting cultural norms (e.g., systems-level supports for mental health and listening to students) and destigmatizing mental health practices. As has been noted by others, REM further offered a space for youth participants to express their contributions, helping them and the adult allies visually recognize and process these impacts (Baker & Johannes, 2013). This finding underscores the importance of including youth voices and perspectives in public health intervention and, with appropriate support, having them lead these efforts in their respective communities (Jenkins et al., 2023).
Importantly, at the close of the REM session, multiple participants identified the powerful effects of the session itself, a finding reported by other researchers using this approach (Nobles et al., 2022). Through this qualitative, participatory approach, participants left the session with a renewed appreciation of the impacts of the Agenda Gap intervention and the ways that their involvement has propelled meaningful shifts in responding to youth mental health in their community. Moreover, including participants external to program implementation provided an opportunity to surface outcomes that may not be immediately evident to those directly involved, building intervention buy-in (Egan et al., 2019), fueling further collective action, and contributing to the sustainability of program impacts (Kollock et al., 2012; Welborn et al., 2019).
Methodological Implications
Capturing the impacts and outcomes of complex public health interventions is challenging, leading to a proliferation of research reporting on individual and proximal short-term outcomes. Often missing in the scientific evidence are the key impacts of longer-term and community-oriented outcomes that account for real-world complexity (Barry, 2003; McGill et al., 2021). The narrow scope of available evidence results, in part, due to methodological challenges in measuring the broader effects of public health interventions over an extended time horizon (Rutter et al., 2017). REM has recently emerged as a qualitative, low-barrier, and participatory approach to evaluation that is helpful in overcoming some of these methodological challenges to elucidate the long-term and multi-level impacts and outcomes of complex public health interventions.
In line with other researchers’ efforts to communicate REM adaptations (Nobles et al., 2022) and recommendations (Creaser et al., 2024) to support and progress the use of this approach in the context of public health intervention evaluation, we offer the following suggestions based on our learnings: • When conducting REM online, include smaller-sized groups (5–10 participants) to ensure adequate time for the process (i.e., partner interviews, story sharing, and broader group reflection). • Combine “theming and rippling” with “in-depth rippling” via reflexive thematic analysis post-event using the session transcript and mind map created during the online session as data sources. This approach saves participants’ time, grounds insights in their narratives, and provides the needed transparency for critique and further analysis. • Include REM facilitators without direct program involvement who can bring curiosity and promote emergent lines of inquiry, alongside research team members with direct intervention involvement who can utilize their experience to support real-time linkage of ideas and concepts. • Use REM as one component of a broader program of evaluation to ensure a comprehensive strategy designed to capture intervention impacts and outcomes over time and across socioecological levels. REM is particularly valuable in illuminating intervention impacts that extend beyond direct intervention participants to influence at the community or population level.
Limitations and Future Research
While our findings lend momentum to the use of REM for evaluating complex public health interventions leading to community or systems change, there are limitations that warrant discussion. First, while REM is powerful for surfacing distal and multi-level ripples, it is not designed to adjudicate mechanisms. Moreover, the use of reflexive thematic analysis on ripple pathways preserves linear sequences but cannot alone establish causal pathways. Future work will therefore need to integrate REM into a broader evaluation program with, for example, social-network analysis of partnerships, follow-up interviews, realist-informed inquiry, and administrative indicators to interrogate the “how/why” alongside “what/where/for whom.” Another limitation of this study is that participants were drawn from one region of British Columbia, Canada, and therefore, their perspectives reflect this context. The particularities of youth mental health and related supports may differ in other settings, as would the anticipated impacts and outcomes of Agenda Gap or other public health interventions for youth mental health. With this in mind, future research is needed to continue to advance the REM approach, including to account for and integrate data from across multiple settings. Indeed, methodological innovation to support REM insights that incorporate the perspectives of participants from across different implementation site would make a considerable contribution to the field. This includes enriching understandings of both the shared and unique intervention impacts for a particular setting alongside the meta-impacts and related mechanisms influencing long-term outcomes across contexts. Additionally, the use of REM and its focus on appreciative inquiry holds the potential to orient participant’s accounts toward more positive intervention outcomes and, as such, can introduce an element of social desirability bias. However, incorporating this strengths-based approach alongside other evaluation modalities helps to ensure a balanced approach, contributing to a robust and holistic understanding of intervention impacts and outcomes over time. A final limitation to acknowledge relates to the three-year interval between program implementation (2020–2021) and REM data collection (2024). While this temporal distance enabled capture of longer-term impacts, it may have contributed to recall difficulties and challenges in attributing observed changes specifically to Agenda Gap, as participants and their communities were exposed to numerous other influences and contextual factors during this period. However, the appreciative inquiry approach and collective mind-mapping process helped mitigate some recall challenges, and the extended follow-up addresses a critical gap in the literature regarding longitudinal evaluation of complex public health interventions.
Conclusion
The present study employed an adapted version of REM to identify the long-term impacts and outcomes of Agenda Gap, a complex, multi-component public health intervention advancing youth mental health through policy engagement. Data generated through the REM process extends the findings of past evaluations, revealing three thematic outcome areas with impacts across socioecological levels. Importantly, REM offered youth and adult participants an opportunity to collectively identify and construct a shared understanding of the program’s long-term impacts related to youth and their families, school systems, and society more broadly. Through the REM process, participants recognized that identified ripples have continued to advance, creating waves and leading to a sea change through shifts in cultural norms and practices supportive of strengthening youth mental health.
Footnotes
Acknowledgments
We would like to thank the communities supporting the delivery of the Agenda Gap program. We would also like to express our appreciation to all of the youth and adult allies who have contributed their wisdom to this research.
Ethical Considerations
Ethical approval for the study was obtained by the University of British Columbia Behavioral Research Ethics Board (H17-001602).
Consent to Participate
All participants provided written informed consent prior to data collection.
Author Contributions
EJ: conceptualization; funding acquisition; analysis; and writing—original draft, review, and editing; TMM: analysis and writing—original draft, review, and editing; LM: project coordination; analysis; and writing—review and editing; CM: data collection; analysis; and writing—original draft, review, and editing; CE: conceptualization and writing—review and editing.
Funding
The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This research was funded by the Community Action Initiative and the Public Health Agency of Canada (1920-HQ-000100). Tier II Canada Research Chair. The funders had no role in the study design, data collection, analysis, or preparation of the manuscript.
Declaration of Conflicting Interests
The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.