Sage Journals: Discover world-class research

Abstract

Ripple effects mapping is a qualitative and participatory method, developed to capture the dynamic nature and interacting elements of an initiative, and its impacts. We present our experiences of using ripple effects mapping to evaluate complex public health initiatives across seven case studies in Bradford, UK. Seven researchers engaged in qualitative reflective practice to capture their individual experiences of using ripple effects mapping within their research and practice; outlining how the method has been adapted and highlighting key reflections and recommendations for implementing ripple effects mapping in the future. We developed ten recommendations, with corresponding strategies, for implementing ripple effects mapping sessions. The recommendations outline how ripple effects mapping can be implemented to improve engagement and anticipate and overcome potential barriers. In doing so, we outline how ripple effects mapping can be used, and adapted, to evaluate various public health initiatives, in research and practice.

Keywords

public health reflective practice ripple effects mapping systems approach

Background

Public health issues, such as obesity, physical inactivity, air pollution, tobacco use and type II diabetes, exist within complex systems whereby networks of interconnected components interact in dynamic ways to produce outcomes that are not readily predictable (Rouse, 2008; Safaei et al., 2021; Salgado et al., 2020). Consequently, combating public health concerns requires complex, adaptive approaches to reduce health inequalities and address modifiable risk factors (Bagnall et al., 2019). This complexity makes evaluating public health initiatives challenging, where impacts may be unpredictable, and manifest via feedback and feedforward loops between interacting components (Moore et al., 2019). Therefore, short-term evaluations may under- or over-estimate impacts (Moore et al., 2019) and solely focusing evaluations on impact-related questions is of limited value. Instead, it is imperative to establish ‘how’ and ‘why’ an initiative works (or does not work) within a complex system, including capturing unintended consequences, to inform future decision-making (Luna Pinzon et al., 2022; Skivington et al., 2021).

Previous research has used ripple effects mapping (REM), alongside other methods, as an evaluation tool to capture an initiative’s intended and unintended consequences (e.g. Hall et al., 2021; King et al., 2021; Luna Pinzon et al., 2022; Nobles et al., 2022a). REM is a qualitative and participatory evaluation method, developed to capture the dynamic nature and interacting elements of an initiative (Nobles et al., 2022b). Introduced by Chazdon et al. (2017), REM sessions incorporate: (1) identifying the intervention, (2) scheduling the event and inviting participants, (3) conducting appreciative inquiry interviews, (4) mapping the ripples and (5) cleaning, coding, and analysing the mind map. The REM process involves producing a visual representation of an initiative via participatory workshops, where stakeholders produce a map of activities, stages, and impacts, capturing the ‘ripples’ leading from one event to another (Chazdon et al., 2017). The process enables stakeholders to articulate their experiences of an initiative, making activities that may otherwise appear intangible, tangible, by explicitly capturing nuances. Stakeholders are also encouraged to identify the most, and least impactful components of an initiative, providing critical insights around the impacts identified (Chazdon et al., 2017; Nobles et al., 2022b). Thus, the reflective nature of REM further develops understanding of the acceptability and feasibility of an initiative and may help future adaptation and implementation of initiative components. Within public health, REM has been used to evaluate the impact of various initiatives, such as undergraduate global health (Bailes et al., 2023), community-based physical activity (PA) (King et al., 2021), and well-being promotion (Spain et al., 2021). Strengths of using REM have been acknowledged, including engaging the community in research, uncovering insights typically unaccounted for with other qualitative methods, and developing and refining future initiatives (Cafer et al., 2022; Duea et al., 2022; King et al., 2021).

The approach to REM by Chazdon et al. (2017) has been adapted and used differently across research projects. One notable example of such adaptation is by Nobles et al. (2022b), where REM was adapted to evaluate a systems initiative aimed at increasing PA in individuals living in Gloucestershire, England; the ‘We Can Move’ initiative (We Can Move, 2021). The paper by Nobles et al. (2022b) outlines five stages to using REM: (1) preparation for the REM workshops, (2) stakeholder recruitment, (3) the initial REM session, (4) follow-up REM session(s) and (5) analysis of the REM outputs. Nobles et al. (2022b) recommend running a series of REM sessions alongside the initiative implementation period. During the initial REM session, a presentation outlining the background to REM, the rationale for its use, and an overview of the REM session is provided (Nobles et al., 2022b). Stakeholders are often divided into groups of three to five people, and the first 10–15 minutes of the REM sessions are allocated to team-based discussions to consider what has been successful within the initiative, their experiences of being part of the initiative, what made achievement or impacts possible, and any unexpected consequences of the initiative (Nobles et al., 2022b). Each group then has approximately 90 minutes to map activities and impacts on a large piece of paper, along a timeline (a new addition to the method originally outlined by Chazdon et al. 2017). Ripples between activities and impacts are depicted using arrows. Impacts are further reflected on by encouraging session stakeholders to consider: (1) who, and how many people, have been impacted upon; (2) whether there has been any financial implications associated with the impacts; (3) if the impacts were intended or unintended; (4) what else may have contributed to these impacts; (5) whether their work links with wider work in the initiative or other organisations/initiatives and (6) if there are any recurring trends being observed across their REM output (Nobles et al., 2022b). Nobles et al. (2022b) also highlighted the potential to use REM prospectively, as well as retrospectively, by encouraging REM stakeholders to map anticipated activities and impacts. Finally, stakeholders can identify and reflect upon the least and most significant changes in the REM outputs, before drawing the session to a close with a reflective conversation with stakeholders. Follow-up REM sessions are also encouraged to continue building the REM outputs considering recent developments (2–3 months recommended between sessions). Follow-up sessions, with the same or additional stakeholders, can further clarify impacts and activities, and update and understand new impacts and activities (Nobles et al., 2022b). The identification of impact pathways (chains of events leading to a cascade of impacts) are recommended to support the analysis of the REM output (Nobles et al. 2022b).

Although REM is well used as an evaluation tool, it is not without its challenges, including low engagement rates (Peterson and Skolits, 2019), difficulties with facilitation capacity (Bloom, 2021) and challenges scheduling REM sessions to include all appropriate stakeholders (Sadeghzadeh et al., 2022). Qualitative reflective practice is a method of self-assessment and personal development through which individuals critically examine their experiences, actions and responses to develop insight and ways to improve practice (Gustafsson et al., 2007). This approach involves engaging in reflections through writing and discussion to explore behaviours and interactions. By identifying patterns and reflecting on decisions practitioners can adapt their strategies. This article is based on qualitative reflective practice of using REM to evaluate complex public health initiatives across seven case studies, and outlines recommendations for conducting REM, which provides practical guidance to inform future use of REM as an evaluative method in research and practice.

Methods

Study design

This research article is based on reflections from a process of qualitative reflective practice to understand experiences and insights (Gustafsson et al., 2007). This approach involves examining the personal experiences of individuals who engage in a specific practice or process through reflective analysis. By capturing thoughts and observations, qualitative reflective practice provides deep, contextual insight into how and why outcomes occur.

Sampling approach

We undertook a convenience sampling approach (Emerson, 2015) in this professional qualitative reflective practice study. This approach permitted researchers from Bradford, UK to capture their reflections on using REM to evaluate complex health initiatives. Inclusion of seven case studies permitted enough breadth and depth to support the development of recommendations. We selected the seven case studies as they covered projects that have used REM with diverse projects and audiences.

About the researchers

We (A.C, L.D, Z.H, J.C, L.C, R.W, E.Y, J.H) present our experiences of using the 2022 adapted version of REM (Nobles et al., 2022b) to evaluate complex public health initiatives, implemented in Bradford, UK. Our roles include research assistant (E.Y, L.C), PhD student (Z.H), research fellow (A.C, J.C) and senior research fellow (L.D, R.W, J.H). We are mostly based within research departments of the National Health Service (NHS) and two of us (L.D, R.W) are embedded researchers in the City of Bradford Metropolitan District Council (CBMDC).

Setting

The City of Bradford, West Yorkshire, is considered one of the youngest and most diverse cities in the United Kingdom with 26.3% of the population aged 18 and under compared to 21.4% nationally (Dogra et al., 2023). Bradford is one of the 20% most deprived areas in England, with 30% of children in low-income families, compared to the national average of 15.3% (City of Bradford Metropolitan District Council, 2020). Prevalence of childhood obesity is also above the national average (22.6% vs 20.5% at age 10–11 years), with academic attainment being lower (Pickett et al., 2021). Numerous research organisations and projects have prioritised ways to improve the health and well-being of the Bradford population. The most notable organisation is the Bradford Institute for Health Research (BIHR), which hosts the Born in Bradford (BiB) research project, which aims to improve the health and well-being of children and their families by collaborating closely with local communities, services, and organisations to make City-wide changes.

REM case studies

We describe seven case studies, outlining our use of REM in the evaluation of complex public health initiatives in Bradford. The case studies are situated within four initiatives: (1) JU:MP (Hall et al., 2021), (2) ActEarly (Mansukoski et al., 2023; Wright et al., 2019), (3) Living Well (City of Bradford Metropolitan District Council, 2021; My Living Well, 2023) and (4) the Adversity, Trauma and Resilience Strategy (Bradford District Adversity, Trauma and Resilience, 2021). We are at different stages of implementing REM (as of October 2023), with most of us currently implementing REM (Z.H, J.C, L.C, R.W, E.Y, J.H), and others having completed all REM sessions (A.C, L.D). We deemed it appropriate to include a range of case studies that reflect distinctively different ways of using REM that we anticipated would yield different, contextually specific insights. Similarly, we intend for this article to provide readers with examples for how REM can further be adapted to meet the needs and contexts of local initiatives and evaluations. Table 1 provides an overview of how each case study has utilised REM to evaluate public health initiatives.

Table 1.

An overview of the REM process across each case study.

Case study	Stage of REM*	Research question(s)/aims REM is addressing	Stakeholders involved in REM workshops	Facilitators
1. JU:MP Leads	Complete	1. What are the activities and impacts of delivering the JU:MP Leads programme? 2. What are the most impactful training and delivery components of the JU:MP Leads programme?	n = an average of five participants per session in phase 1, and 15 per session in phase 2; young people, aged 16–25 years (JU:MP Leads) and LTO representatives, including volunteers and professionals	Sessions were delivered by (J.H), between 2020 and 2022 (phase 1), and (J.C), between 2022 and 2023 (phase 2)
2. JU:MP strategic influencing	In progress	What is the impact of strategic influencing work in Bradford related to physical activity and related change mechanisms?	n = an average of 10 participants per session, who typically attend all REM sessions; key delivery stakeholders who are senior professionals in public health, sport and physical activity. They are part of a PA strategy group, invited to participate in the research on this basis	Sessions were delivered by (J.H), from 2019 to 2022, with input from (R.W and E.Y), This role was handed to (J.C) in 2023
3. JU:MP neighbourhoods	In progress	1. What activities and impacts have occurred in relation to the work of JU:MP’s hyper-local ‘neighbourhood’ approach (including wider intended/unintended)? 2. How might these activities and impacts differ according to neighbourhood context (what works, for whom, and how)?	n = an average of 10 participants per session per neighbourhood (four neighbourhoods in total); key delivery stakeholders who work within the neighbourhood, such as school and faith settings stakeholders, voluntary and community sectors organisations, activity providers and members of core JU:MP team (e.g. community engagement manager, researcher)	(J.H) leads the JU:MP neighbourhood process evaluation and (E.Y) is responsible for delivering the commissioned neighbourhood evaluation
4. Creating active schools (CAS)	In progress	1. How is CAS being implemented within Bradford? 2. What elements of CAS are more widely engaged with and what has the greatest impacts on changes to physical activity provision? 3. How does the effectiveness of CAS differ between schools with high, medium or low implementation?	n = an average of five participants per session, with anomalies, including 2 or 23 (due to staffing issues, change in school needs, etc.); school staff, such as senior leadership team, CAS champions, in-school CAS leads, teachers, governors and wider school staff	Sessions are delivered by (Z.H and L.C) and facilitated or observed by research assistants and placement year students
5. ActEarly	Complete	Has ActEarly influenced local and national decision-making and if so, how?	n = an average of 6 participants per session; ActEarly staff members, including administrative staff, research fellows, principal investigators and members from partnering organisations	Sessions were delivered by a research assistant, with assistance from other members of the ActEarly evaluation team
6. Living Well	In progress	To develop a collaborative understanding of progress, activities and impact taking place in Living Well initiatives across various levels of the local system: (1) physical environment, (2) communities and organisations, (3) systems level – strategic development	n = an average of 7 participants per session; public health professionals working in the local authority, including senior leaders and partner organisations	Sessions are delivered by (R.W) with input from (Z.H) and (L.D)
7. Adversity, trauma and resilience (ATR) strategy	Complete	To understand how strategic work related to adversity, trauma and resilience has been employed with a focus on successes, challenges and gaps in implementation to inform a renewed strategy	n = 25; members of the ATR team and core working groups	Session was delivered by (L.D)

Cases 1–4: JU:MP – Bradford Local Delivery Pilot

JU:MP, funded by Sport England between 2019 and 2025, is one of 12 local delivery pilots (LDPs) across England taking a whole systems approach to increase PA and reduce health inequalities within a locality and create sustainable change (Hall et al., 2021). The JU:MP initiative targets children aged between 5 and 14 years, and their families, in the North of Bradford. A key characteristic of JU:MP is that it is flexible and adaptive, to allow local children, families, and organisations to take ownership of the implementation, which both ensures that it is tailored to local need and context, while facilitating sustainability and longer-term behavioural change (O’Mara-Eves et al., 2015). JU:MP comprises 15 workstreams that cut across five themes: city-wide policy and strategy, community, organisations, environment and families (Hall et al., 2021). We present four case studies, making up four of the 15 JU:MP workstreams. The process evaluation of JU:MP is underpinned by realist principles, and uses mixed-methods data collection approaches, such as semi-structured interviews, observations, documentary analysis, surveys and REM (Frazer et al., 2023; Hall et al., 2021).

Case 1: JU:MP Leads

JU:MP Leads is a commissioned sport and PA leadership initiative for young people aged 16–25 years (Hall et al., 2021). The workstream supports young people to deliver informal sport/PA and includes training packages and community placements (Hall et al., 2021). JU:MP Leads aims to mobilise community assets through capacity building and improve social mobility of young people from deprived communities (Berrie et al., 2023). The local authority delivered phase 1 of JU:MP Leads. Twelve local young people participated over a 15-month period (2020–2022). Phase 2 was delivered by StreetGames (2022–2023) and involved 20 young people. StreetGames were responsible for overarching delivery and training provision, with one of four local trusted organisations (Sunnah Sports CIC, Greater Horton Church Cricket Club, Bradford Foundation Trust and Women Zone) mentoring the young people and providing delivery experience.

Case 2: JU:MP strategic influencing

Living Well (City of Bradford Metropolitan District Council, 2021) and Active Bradford (2020) deliver the ‘strategic influencing’ workstream in partnership. A strategic development group meets regularly to develop and monitor plans to embed PA within policy and strategy across the Bradford district (Hall et al., 2021). The JU:MP process evaluation includes an examination of the acceptability, feasibility and impact of JU:MP on city-wide policy and strategic working around PA (Hall et al., 2021).

Case 3: JU:MP neighbourhood approach

JU:MP is being operationalised through a hyper-local neighbourhood approach, including eight geographic neighbourhoods, in Bradford (Hall et al., 2021). Two neighbourhoods were commissioned to the local authority to deliver, and six neighbourhoods were facilitated by a community engagement manager who worked directly for JU:MP. A JU:MP Action Group (JAG) was established within each neighbourhood to co-produce and deliver JU:MP in the neighbourhood. JAG stakeholders developed and implemented a local action plan. A structured process was followed to facilitate an evidence-informed approach to neighbourhood delivery, which included incorporating behaviour change principles and rapid evidence reviews into action plan development. REM formed part of a neighbourhood process evaluation, which aimed to examine the acceptability and feasibility of the neighbourhood-level design and implementation of the JU:MP initiative, by understanding the barriers, facilitators and contextual factors influencing design and delivery (Hall et al., 2021). REM was conducted at six-monthly intervals.

Case 4: Creating active schools

Creating active schools (CAS) is a novel UK-based whole-school approach to PA that is underpinned by behavioural and implementation science (Daly-Smith et al., 2020; Damschroder et al., 2009). The programme comprises a four-stage annual cycle (review, action planning, implementing, monitoring and evaluating) to achieve sustainable organisational-level change across four specific areas: policy, environments, stakeholders and opportunities, across the school day and beyond (Daly-Smith et al., 2020). CAS champions support schools and facilitate engagement, check and challenge schools, support the development and implementation of CAS and connect schools with locality-based communities of practice (Helme et al., 2022). The evaluation of CAS is incorporated into the JU:MP evaluation (Hall et al., 2021), as well as having its own evaluation (Morris et al., preprint). REM is being used alongside accelerometery data to understand the interaction between implementation and effectiveness of CAS. Through REM, schools that are high, medium and low implementers (through understanding what of the initiative and how much of the initiative they have engaged with and implemented) can be identified. Comparisons can be made to change in PA levels in children attending CAS schools, derived from the accelerometery data.

Case 5: ActEarly

ActEarly, funded by the UK Prevention Research Partnership (UKPRP), is a collaborative consortium aimed at improving the health and well-being of children and families living in two areas with high levels of deprivation: Bradford, West Yorkshire and the London Borough of Tower Hamlets (Wright et al., 2019). ActEarly works across seven interconnected themes: (1) healthy livelihoods, (2) healthy learning, (3) healthy places, (4) food and healthy weight, (5) play and PA, (6) citizen science and co-production and (7) evaluation (Wright et al., 2019). ActEarly has implemented over 20 interventions and 40 activities (research, data initiatives, citizen science and community engagement), between September 2019 and December 2022 (Mansukoski et al., 2023). A meta-evaluation (evaluation of evaluations) of ActEarly is currently ongoing, and involves using mixed-methods approaches (e.g. quantitative: systems mapping, natural- and quasi-experimental evaluations, data visualisations, computer simulation and qualitative: interviews, documentary analysis) (Mansukoski et al., 2023).

Case 6: Living well

Living Well is an initiative led by the CBMDC Public Health and Bradford District and Craven Health and Care Partnership aimed at addressing rising levels of obesity and reducing early and preventable deaths within Bradford (City of Bradford Metropolitan District Council, 2021). Living Well is striving to harness the potential of the local system and capitalise on the assets of the district to deliver work ‘at scale’ to address these issues. The approach enables behaviour change by: (1) targeting leadership, governance, partnerships and capabilities across the system to develop and implement coordinated actions; (2) modifying the physical environment and ensuring people of all ages have equitable access to health promoting places and spaces; (3) creating an asset-based approach and facilitating adjustments to policies and practices in communities, businesses and organisations and (4) creating and promoting access to information, advice and support to help individuals, families and communities engage in living healthier lifestyles (City of Bradford Metropolitan District Council, 2021; My Living Well, 2023). A combination of methods are being utilised to evaluate Living Well, including systems mapping, social network analysis, REM and service evaluation. This aims to establish how Living Well is impacting the system, both in terms of shifting the behaviours of system partners and actions by others in the system changing the behaviours of Bradford communities.

Case 7: Adversity, trauma and resilience strategy

The ATR strategy (2021–2024) is led by CBMDC Public Health team, in partnership with Better Start Bradford, Children’s Social Care, and West Yorkshire Health and Care Partnership ATR initiative (Bradford District Adversity, Trauma and Resilience, 2021). The ATR strategy aims to reduce the impact of adverse childhood experiences and trauma on the population across the Bradford District, build resilience against long-term harm of adversity and trauma, and raise awareness and improve access to support across the lifespan (Bradford District Adversity, Trauma and Resilience, 2021). To address these aims, the strategy comprises an ATR steering group and four working groups: communities, workforce, early years and education (Bradford District Adversity, Trauma and Resilience, 2021). The main purpose of REM in this case study was to provide stakeholders an opportunity to reflect on the strategy and develop plans for the future iteration. The evaluation of this project is not deemed ‘research-led’ and used to support practice.

Data collection and analysis

We drew on a reflective practice process outlined by Potts et al. (2022) to collect, analyse and critique data about the REM process and adaptations made to the Nobles et al. (2022b) REM method across the case studies. As this article is based on qualitative reflective practice, data collection and analysis occurred iteratively and concurrently:

Step 1: All authors met to discuss the purpose of outlining case studies and agreed on what information we would collect.

Step 2: We each developed a case study (A.C, L.D, Z.H, J.C, L.C, R.W, E.Y, J.H) by independently completing a table outlining how we used/intended to use REM. This table included the following dimensions related to REM: rationale, context, stakeholders and facilitators, delivery setting, adaptations, analysis. We each also produced a document outlining their reflections of using REM and recommendations for future research.

Step 3: We met to discuss the case studies and reflections. Two other authors reviewed each case study and provided critical feedback on the step 2 documentation, such as highlighting areas where more clarity, or elaboration on key points, was required.

Step 4: We each refined our reflective case studies based on the critical feedback.

Step 5: A.C, L.D, J.C and L.C collectively reviewed the recommendations for using REM and began to synthesise and organise the recommendations by identifying similarities and differences across case studies. We organised the findings using post-it notes, where we grouped similar recommendations together, and considered what the recommendation best reflected.

Step 6: L.C considered ways to present the recommendation clusters, and presented options for A.C, L.D and J.C to provide feedback. A.C, L.D, J.C and L.C retrospectively mapped reflections of using REM with the recommendations outlined, considering what recommendations and strategies may overcome barriers of using REM for prospective users. We all reviewed the recommendations and, following revisions, agreed upon the final content.

Results

Adaptations of the REM method

Through the analysis, we identified several adaptations to REM that were needed to apply the REM method to various settings, projects and audiences. We divided these into three themes: (1) adaptions made during REM session(s), (2) adaptations made after initial REM session(s) and (3) Adaptations to REM analysis. We provide an outline of these adaptations and the underpinning rationale(s) below.

Adaptions made during REM session(s)

We made most adaptations during the REM sessions, largely, at least in part, for time-saving purposes. Two case studies omitted the team-based discussion as time constraints made team-based discussion not a priority, and we embedded REM in team meetings where such discussions already took place. In four case studies, we mapped key events prior to REM sessions, due to time constraints and to provide a ‘springboard’ for discussion during the session. We encouraged stakeholders to move backwards as well as forwards in time on the timeline in six case studies as this facilitated reflection and offered a flexible, stakeholder-led process. The CAS case study replaced the ‘most and least significant changes’ section with ‘barriers and facilitators of implementing the initiative’ as this better supported the process of adapting the initiative following REM.

To make the process more accessible for young people, researchers in the JU:MP Leads case study initially mapped events mentioned in team-based discussions and young people then built on this. In addition, as multiple young people participated with different levels of engagement, we did not discuss changes in a group context, to avoid (negative) comparisons between leaders who experienced different journeys.

Adaptions made after initial REM session(s)

Four case studies added clarification or additional information to the map, taken from interviews, informal conversations with stakeholders or REM transcripts, and two case studies circulated the maps for stakeholders to review and add to following the session(s). These strategies permitted deeper analysis and supported accuracy.

Analysis of the REM output(s)

Within the case studies, we used different methods to analyse the REM outputs, which align with each initiative’s aims. The CAS and strategic influencing case studies adopted a deductive framework approach to analysis. We deemed this approach useful for enabling researchers to draw comparisons between a large number of maps at one time and understand similarities and differences between settings (e.g. schools, neighbourhoods) following implementation of the initiative. A framework approach also enables analysis of the ripple maps in conjunction with data from other sources (e.g. observations, interviews), and it allows multiple researchers to contribute to the analysis. The ActEarly case study employed an inductive thematic analysis as we deemed this suitable to feeding into the development of context–mechanism–outcome (CMO) configurations as part of the realist approach. We did not conduct a formal analysis within the Living Well case study as we primarily delivered REM to inform practice, that is, collective learning and setting future priorities. As such, we felt that formal analysis would not be timely or informative.

Recommendations

We developed ten recommendations through qualitative reflective practice (see Figure 1). Although the recommendations are presented separately, they are interrelated and context-dependent. Thus, the recommendations should be considered ‘principles’ and useful starting points to optimise session engagement, delivery and output, rather than set actions to undertake when conducting REM. Table S1 displays the developed recommendations, corresponding strategies and reflections in full.

Figure 1.

Ten Recommendations for implementing Ripple Effects Mapping (REM).

Discussion

The aim of this qualitative reflective practice paper was outline, through case studies, how REM can be used, and adapted, to evaluate various public health initiatives, and develop recommendations and strategies for implementing REM in future practice. All seven case studies in the present study made adaptations to the following domains: length of sessions, timings of the sessions, number of stakeholders, delivery method (face-to-face/online) and nature of stakeholders involved. We typically made adaptations to ensure REM aligned with the initiative needs and aims, as well as in response to external factors (e.g. time constraints) that impacted delivery. This evidences the malleable nature of REM and its ability to flex depending on initiative needs to ensure REM outputs illustrate changes to their own complex system and identify both the intended and unintended outcomes (Nobles et al., 2022b). Future research should aim to evaluate the REM method and understand if such adaptations have an impact on its application, acceptability and feasibility.

Our collective reflections of using REM helped to form ten recommendations for those seeking to adopt REM in future (see Figure 1). Researchers should determine the aim(s) of conducting REM, and whether additional evaluation methods (e.g. interviews) will be used alongside REM (recommendation 1). We recommend producing a protocol, before conducting REM, which includes the research question(s) REM is addressing, as well as how REM findings will support other methods. It may also be useful to outline prompts for use during the REM sessions to help conversations meet the aims of REM. We also recommend establishing the boundaries of the initiative and the evaluation (recommendation 2). While establishing rigid boundaries of a complex initiative is difficult and can sometimes be arbitrary for the purpose of initiative evaluation (Luna Pinzon et al., 2022), outlining flexible initiative boundaries can be important for ensuring implementation of REM meets the evaluation aims. Whole system frameworks encourage boundaries to be defined (e.g. the ‘Evaluation of initiatives in Complex Adaptive Systems (ENCOMPASS)’ framework; Luna Pinzon et al., 2022). Strategies to establish boundaries include considering an initiative’s purpose and/or who and what is part of the initiative/system of interest (Foster-Fishman et al., 2007).

Within the seven case studies presented, we made key adaptations to the REM method due to external factors that impacted the delivery of REM (e.g. time constraints). It can often be difficult to anticipate these, however, we recommend having an awareness of the physical space, online resources, and amount of time allocated to the REM session(s) (recommendations 3 and 4). Creating an agenda (e.g. supplementary file 2) that outlines the timings, space and resources needed to conduct REM is useful. However, having a contingency plan and being flexible can also overcome any unanticipated changes to the REM session. When external factors can be anticipated, we recommend making appropriate adaptations to the REM method (recommendation 5). Strategies that may help with time efficiency included adding known activities/events onto the REM timeline prior to the session or providing an overview of REM prior to the session (see Supplementary Material 3, for example, of an overview of REM provided to stakeholders).

Several recommendations consider the importance of knowing and engaging REM stakeholders (recommendations 6–10). We deem engaging a range of individuals in research to be imperative to translating research into practice (Handley et al., 2016). Building trusted relationships between stakeholders, researchers and the researcher’s institution is considered pivotal to successful research (Guillemin et al., 2018). To practically implement REM, it is important to know the target audience (recommendation 6), which can impact the degree of formality used during the REM sessions. In some case studies (e.g. JU:MP Leads), we deemed it appropriate to create an informal environment by providing food and beverages (target audience: 16- to 25-year-olds), whereas in other case studies (ActEarly), we deemed it appropriate to make the session as formal and ‘research-driven’ as possible (target audience: professionals and academics). Knowing the REM audience can also influence who is the most appropriate person to deliver sessions (recommendation 7). Existing knowledge of an initiative’s context, stakeholders and relationships is key when implementing research methods to evaluate complex initiatives (Luna Pinzon et al., 2022; Moore et al., 2019), therefore, it may be useful to consider a person’s ability to meet the aims of REM and engage stakeholders. For example, in the JU:MP ‘s ‘neighbourhood’ case study, implementers of the initiative (non-researchers) and researchers, collectively delivered REM to encourage the sustainability of using REM in practice. It may be appropriate to formally train others to deliver REM; in which case utilising existing REM training materials would be useful (e.g. the National Institute for Health Research (NIHR) Applied Research Collaboration training resources).

We recommend that researchers/evaluators engage stakeholders to increase ‘buy-in’ and build trust (recommendation 8). Strategies that may improve ‘buy-in’ include delivery of REM sessions by individuals the stakeholders are familiar with and implementing sessions into pre-existing meetings/events. Previous research has found that teachers felt time was a limiting factor preventing educators from using research in their decision-making and practices (Joram et al., 2020). Therefore, by making the REM sessions more accessible for stakeholders, the sessions may be less burdensome. Ensuring REM is reciprocally beneficial for all involved (recommendation 9) may build trust, sustain engagement with the method and enable the benefits of REM to extend beyond the research, and used in practice. In the CAS and Living Well case studies, school staff and council members have used the REM method in practice, enabling them to monitor progress within their role, as well as within the research. Strategies included being flexible in the approach to the REM process, enabling stakeholders to use the REM output/map that works for them, and enabling stakeholders to set priorities within the initiative. Implementing REM sessions in such a way can increase engagement with the method. In two case studies (Living Well and ATR strategy), council members actively sought out the use of REM based on other colleagues’ experience with REM. This highlights REM’s ability to lessen the so-called ‘research-practice gap’ (Bansal et al., 2012), and how REM may start to create ‘ripples’ to the implementation of an initiative itself (Nobles et al., 2022b).

Our final recommendation for using REM is to consider power imbalances and influence on REM engagement and outputs (recommendation 10). Social dynamics/hierarchies between stakeholders can impact engagement with REM, such as some stakeholders having less opportunity to contribute than others. One strategy that may alleviate this is to split stakeholders into smaller groups, and purposely select who is in each group. We recommend that, where possible, additional facilitators are present during the REM session, who can observe interactions between stakeholders (e.g. using an observation sheet; supplementary file 4). Although this will not lessen power dynamics, capturing such interactions may be beneficial when interpreting and/or disseminating findings. It may also be useful to consider power dynamics between the researcher/evaluator(s) and stakeholders (Van der Riet and Boettiger, 2009) when deciding who delivers the REM session(s).

Future directions for REM

REM is now being widely adopted in the public health sphere to help understand how complex initiatives work within complex systems to address wicked problems. It is a pragmatic, practical and malleable method, which we have evidenced herein across seven case studies, and also more widely (e.g. Hall et al., 2021; King et al., 2021; Luna Pinzon et al., 2022; Nobles et al., 2022a). Anecdotally, REM is fulfilling a methodological gap. While it is encouraging to see the adoption of a method, such as REM, it is important for future research to examine the utility of REM. Considering the strengths and limitations of the method will provide a stronger foundation on which to continue applying and refining REM. This article provides a first step towards this, but more substantive work is required to build the evidence-base around REM, and to consider the perspectives of those involved in REM workshops, those using REM to inform their work, and those funding REM-related projects. Future research should also aim to understand how REM can be used alongside quantitative methods to support initiative evaluation (Hall and Mansfield, 2023), and the interplay between implementation processes and effectiveness.

Strengths and limitations

This article is the first to outline how the REM method can be applied and adapted to evaluate various complex public health initiatives, and outline recommendations for implementing REM in the future. We demonstrate the versatility of using REM, with the seven case studies engaging different stakeholders, across different time periods, with different aims and priorities, and analyses. Outlining how REM was used in these case studies, as well as outlining recommendations and strategies for implementing REM, can improve the applicability, and use of REM in future research and practice. A potential limitation was our focus on one geographical area. We encourage reflections from researchers further afield to add to the recommendations in this study. In addition, all case studies primarily used REM for research purposes. However, two of us (L.D and R.W) are embedded researchers in the local council, which highlights the use of REM in practice, and we have reflected on the benefits of using REM beyond addressing research aims (e.g. CAS). A further limitation of this article is that insights and reflections from REM stakeholders and other non-researcher deliverers/facilitators were not collected to inform the recommendations produced in this article. We welcome further reflections of using REM from these individuals to further expand and adapt learning.

Conclusion

The aim of this article was to present case studies outlining how REM has been applied and adapted to evaluate complex public health initiatives in Bradford, UK. REM has emerged as a powerful tool to meet an ever-intensifying evaluation challenge, documenting the impacts of such initiatives within real-life settings (Chazdon et al., 2017). Through engaging in qualitative reflective practice, we have lent into the adaptable nature of REM to identify ten recommendations for applying this method to inform future evaluations in research and practice. We were able to demonstrate how REM can be used to evaluate a variety of initiatives, engaging different stakeholders, across different time periods, and with different aims and priorities. By showcasing how the method can flex to meet the needs of multiple initiatives, stakeholders and contexts, this article can inform the future application of REM. To improve the experiences of conducting REM, future initiatives should consider the adaptations and recommendations elicited in this article within their evaluation plan.

Supplemental Material

sj-docx-1-evi-10.1177_13563890241285032 – Supplemental material for Adapting ripple effects mapping for evaluating public health initiatives in complex systems: Reflections and recommendations from seven case studies

Supplemental material, sj-docx-1-evi-10.1177_13563890241285032 for Adapting ripple effects mapping for evaluating public health initiatives in complex systems: Reflections and recommendations from seven case studies by Amy Creaser, Lisa Dowling, Zoe Helme, Jamie Crowther, Lily Casana, Rachel Williams, Emma Young, James Nobles and Jennifer Hall in Evaluation

Supplemental Material

sj-docx-2-evi-10.1177_13563890241285032 – Supplemental material for Adapting ripple effects mapping for evaluating public health initiatives in complex systems: Reflections and recommendations from seven case studies

Supplemental material, sj-docx-2-evi-10.1177_13563890241285032 for Adapting ripple effects mapping for evaluating public health initiatives in complex systems: Reflections and recommendations from seven case studies by Amy Creaser, Lisa Dowling, Zoe Helme, Jamie Crowther, Lily Casana, Rachel Williams, Emma Young, James Nobles and Jennifer Hall in Evaluation

Supplemental Material

sj-docx-3-evi-10.1177_13563890241285032 – Supplemental material for Adapting ripple effects mapping for evaluating public health initiatives in complex systems: Reflections and recommendations from seven case studies

Supplemental material, sj-docx-3-evi-10.1177_13563890241285032 for Adapting ripple effects mapping for evaluating public health initiatives in complex systems: Reflections and recommendations from seven case studies by Amy Creaser, Lisa Dowling, Zoe Helme, Jamie Crowther, Lily Casana, Rachel Williams, Emma Young, James Nobles and Jennifer Hall in Evaluation

Supplemental Material

sj-docx-4-evi-10.1177_13563890241285032 – Supplemental material for Adapting ripple effects mapping for evaluating public health initiatives in complex systems: Reflections and recommendations from seven case studies

Supplemental material, sj-docx-4-evi-10.1177_13563890241285032 for Adapting ripple effects mapping for evaluating public health initiatives in complex systems: Reflections and recommendations from seven case studies by Amy Creaser, Lisa Dowling, Zoe Helme, Jamie Crowther, Lily Casana, Rachel Williams, Emma Young, James Nobles and Jennifer Hall in Evaluation

Footnotes

Acknowledgements

The authors would like to acknowledge the stakeholders and participants who have been involved in the REM workshops to make this paper possible. They would also like to thank members of the ActEarly evaluation team (Laura Nixon, Liina Mansukoski, Patience Gansallo and Louise Padgett) who delivered and supported the ActEarly REM sessions.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship and/or publication of this article: All the studies that are presented as case studies in this paper received funding from the following sources: JU:MP – The authors’ involvement was supported by Sport England’s Local Delivery Pilot – Bradford and the National Institute for Health Research (NIHR) Yorkshire and Humber Applied Research Collaboration (ARC). Sport England is a non-departmental public body under the Department for Digital, Culture, Media and Sport (DCMS).

Creating Active Schools – The authors’ involvement is supported by a match funded PhD by the University of Bradford and Sport England Local Delivery Pilot Bradford.

ActEarly and ATR – The authors’ involvement is supported by the UK Prevention Research Partnership (MR/S037527/1), which is funded by the British Heart Foundation, Cancer Research UK, Chief Scientist Office of the Scottish Government Health and Social Care Directorates, Engineering and Physical Sciences Research Council, Economic and Social Research Council, Health and Social Care Research and Development Division (Welsh Government), Medical Research Council, National Institute for Health Research, Natural Environment Research Council, Public Health Agency (Northern Ireland), the Health Foundation and Wellcome.

Living Well – The authors’ involvement is supported by Living Well, a joint initiative led and funded by Public Health at City of Bradford Metropolitan District Council and Bradford District and Craven Health and Care Partnership.

ORCID iDs

Amy Creaser

Jennifer Hall

Research data

All materials related to the REM method are provided in the manuscript and online supplementary materials.

Supplemental material

Supplemental material for this article is available online.

Amy Creaser is a research fellow of the First 1000 Days Project, an initiative aimed at improving health and social outcomes for families, and children, in the first 2 years of life.

Lisa Dowling is a Senior Research Fellow working with ActEarly and the City of Bradford Metropolitan District Council Public Health Team. Her work aims to support evaluation in local government decision-making.

Zoe Helme is a PhD student. Her work focuses on evaluating the implementation of the Creating Active Schools Framework and the impact this has on whole-school physical activity.

Jamie Crowther is a qualitative research fellow at Born in Bradford. His research interests include youth development, violence and masculinities, and sport criminology.

Lily Casana undertook a research internship with Born in Bradford, as part of her undergraduate psychology degree at the University of Leeds.

Rachel Williams leads the embedded research and evaluation for Living Well, Bradford’s whole systems approach to obesity and well-being.

Emma Young is a PhD student. Her work focuses on exploring the relationship between mental wellbeing, weight and eating behaviour using data from the Gemini twin study.

James Nobles is a pragmatic, mixed-methods researcher with specialist expertise in the prevention and treatment of obesity. His core interests lie within the design, implementation and evaluation of systems approaches.

Jennifer Hall is a qualitative health researcher and co-leads the Bradford Centre for Qualitative Research. Her research interests include physical activity, dance and well-being, and health inequalities.

References

Active Bradford (2020). Available at: https://www.activebradford.com/ (accessed 10 July 2023).

Bagnall

Radley

Jones

, et al. (2019) Whole systems approaches to obesity and other complex public health challenges: A systematic review. BMC Public Health 19: 1–14.

Bailes

Haller

Moseley

(2023) Utilizing ripple effects mapping to assess the impact of an undergraduate global health program. Annals of Global Health 89(1): 3933.

Bansal

Bertels

Ewart

, et al. (2012) Bridging the research–practice gap. Academy of Management Perspectives 26: 0140.

Berrie

Adair

Williamson

, et al. (2023) Youth organizations, social mobility and health in middle age: Evidence from a Scottish 1950s prospective cohort study. European Journal of Public Health 33(1): 6–12.

Bloom

(2021) ‘Spiraling up’: Using ripple effect mapping to evaluate how an extension volunteer program increases community development capacity. Community Development 52(1): 22–41.

Bradford District Adversity, Trauma and Resilience (2021) ACEs, trauma & resilience Strategy 2021-2024. Available at: https://www.bradfordatrpartnership.co.uk/media/iqxlnpbe/2021-03-04-atr-strategy-2021-24-zy-v1.pdf (accessed 10 July 2023).

Cafer

Green

Goreham

(2022) A community resilience framework for community development practitioners building equity and adaptive capacity. In: Brennan

Phillips

Walzer

, et al (eds) Community Development for Times of Crisis. London: Routledge, 56–74.

Chazdon

Emery

Hansen

, et al. (2017) A Field Guide to Ripple Effects Mapping. Minneapolis, MN: University of Minnesota Libraries Publishing.

10.

City of Bradford Metropolitan District Council (2020) Understanding bradford district, intelligence bulletin: Poverty and deprivation. Available at: https://ubd.bradford.gov.uk/media/1580/poverty-and-deprivation-jan-2020-update.pdf (accessed 5 June 2023).

11.

City of Bradford Metropolitan District Council (2021) Bradford council plan priorities and principles 2021–2025. Available at: https://www.bradford.gov.uk/media/6508/bradfordcouncilplan.pdf (accessed 5 April 2023).

12.

Daly-Smith

Quarmby

Archbold

, et al. (2020) Using a multi-stakeholder experience-based design process to co-develop the creating active schools framework. International Journal of Behavioral Nutrition and Physical Activity 17: 13.

13.

Damschroder

Aron

Keith

, et al. (2009) Fostering implementation of health services research findings into practice: A consolidated framework for advancing implementation science. Implementation Science 4(1): 50.

14.

Dogra

Lightfoot

Kerr

, et al. (2023) Born in Bradford Age of Wonder cohort: A protocol for qualitative longitudinal research [version 3; peer review: 2 approved]. Wellcome Open Research 7: 270.

15.

Duea

Zimmerman

Vaughn

, et al. (2022) A guide to selecting participatory research methods based on project and partnership goals. Journal of Participatory Research Methods 3(1): 1035844001c32605.

16.

Emerson

(2015) Convenience sampling, random sampling, and snowball sampling: How does sampling affect the validity of research? Journal of Visual Impairment & Blindness 109(2): 164–8.

17.

Foster-Fishman

Nowell

Yang

(2007) Putting the system back into systems change: A framework for understanding and changing organizational and community systems. American Journal of Community Psychology 39: 197–215.

18.

Frazer

Seims

Tatterton

, et al. (2023) Child and family experiences of a whole-systems approach to physical activity in a multiethnic UK city: A citizen science evaluation protocol. BMJ Open 13(2): e069334.

19.

Guillemin

Barnard

Allen

, et al. (2018) Do research participants trust researchers or their institution? Journal of Empirical Research on Human Research Ethics 13(3): 285–94.

20.

Gustafsson

Asp

Fagerberg

(2007) Reflective practice in nursing care: Embedded assumptions in qualitative studies. International Journal of Nursing Practice 13(3): 151–60.

21.

Hall

Mansfield

(2023) The benefits and complexities of integrating mixed method findings using the Pillar Integration Process: A workplace health intervention case study. Journal of Mixed Methods Research. Epub ahead of print 17 August. DOI: 10.1177/15586898231196287.

22.

Hall

Bingham

Seims

, et al. (2021) A whole system approach to increasing children’s physical activity in a multi-ethnic UK city: A process evaluation protocol. BMC Public Health 21(1): 2296.

23.

Handley

Gorukanti

Cattamanchi

(2016) Strategies for implementing implementation science: A methodological overview. Emergency Medicine Journal 33: 660–664.

24.

Helme

Morris

Nichols

, et al. (2022) Assessing the Impacts of Creating Active Schools on Organisational Culture for Physical Activity. International Journal of Environmental Research and Public Health 19(24): 16950.

25.

Joram

Gabriele

Walton

(2020) What influences teachers’ ‘buy-in’ of research? Teachers’ beliefs about the applicability of educational research to their practice. Teaching and Teacher Education 88: 102980.

26.

King

Campero

Garcia

, et al. (2021) Testing the effectiveness of community-engaged citizen science to promote physical activity, foster healthier neighborhood environments, and advance health equity in vulnerable communities: The Steps for Change randomized controlled trial design and methods. Contemporary Clinical Trials 108: 106526.

27.

Luna

Pinzon A

Stronks

Dijkstra

, et al. (2022) The ENCOMPASS framework: A practical guide for the evaluation of public health programmes in complex adaptive systems. International Journal of Behavioral Nutrition and Physical Activity 19(1): 33.

28.

Mansukoski

Lockyer

Creaser

, et al. (2023) Meta-evaluation of a whole systems programme, ActEarly: A study protocol. PLoS ONE 18(6): e0280696. https://doi.org/10.1371/journal.pone.0280696

29.

Moore

Evans

Hawkins

, et al. (2019) From complex social interventions to interventions in complex social systems: Future directions and unresolved questions for intervention development and evaluation. Evaluation 25(1): 23–45.

30.

Morris

Chalkley

Helme

, et al. (preprint). Initial insights into the impact and implementation of Creating Active Schools in Bradford, UK. Available at: https://doi.org/10.21203/rs.3.rs-2458671/v2

31.

My Living Well (2023) Available at: https://mylivingwell.co.uk/ (accessed 10 July 2023).

32.

Nobles

Fox

Inman-Ward

, et al. (2022a) Navigating the river (S) of systems change: A multi-methods, qualitative evaluation exploring the implementation of a systems approach to physical activity in Gloucestershire, England. BMJ Open 12(8): 063638.

33.

Nobles

Wheeler

Dunleavy-Harris

, et al. (2022b) Ripple effects mapping: Capturing the wider impacts of systems change efforts in public health. BMC Medical Research Methodology 22(1): 72.

34.

O’Mara-Eves

Brunton

Oliver

, et al. (2015) The effectiveness of community engagement in public health interventions for disadvantaged groups: A meta-analysis. BMC Public Health 15: 129.

35.

Peterson

Skolits

(2019) Evaluating unintended program outcomes through Ripple Effects Mapping (REM): Application of REM using grounded theory. Evaluation and Program Planning 76: 101677.

36.

Pickett

Taylor-Robinson

Erlam

(2021) Child of the North: Building a fairer future after COVID-19. Project Report. Northern Health Science Alliance and N8 Research Partnership. Available at: chrome-extension://efaidnbmnnnibpcajpcglclefindmkaj/ https://www.thenhsa.co.uk/app/uploads/2022/01/Child-of-the-North-Report-FINAL-1.pdf (accessed 5 June 2023).

37.

Potts

Nobles

Shearn

, et al. (2022) Embedded researchers as part of a whole systems approach to physical activity: Reflections and recommendations. Systems 10(3): 69.

38.

Rouse

(2008) Health care as a complex adaptive system: Implications for design and management. Bridge-Washington-national Academy of Engineering 38(1): 17.

39.

Sadeghzadeh

Sheppard

de Groot

, et al. (2022) Evaluating the benefits of a SNAP-Ed-funded community garden intervention using ripple effect mapping. Health Education & Behavior 49(1): 141–9.

40.

Safaei

Sundararajan

Driss

, et al. (2021) A systematic literature review on obesity: Understanding the causes & consequences of obesity and reviewing various machine learning approaches used to predict obesity. Computers in Biology and Medicine 136: 104754.

41.

Salgado

Madureira

Mendes

, et al. (2020) Environmental determinants of population health in urban settings. A systematic review. BMC Public Health 20: 853.

42.

Skivington

Matthews

Simpson

, et al. (2021) A new framework for developing and evaluating complex interventions: Update of Medical Research Council guidance. BMJ 374: 2016.

43.

Spain

Stewart

Betts

, et al. (2021) Wheel of Wellbeing (WoW) health promotion program: Australian participants report on their experiences and impacts. BMC Public Health 21: 2037.

44.

Van der Riet

Boettiger

(2009) Shifting research dynamics: Addressing power and maximising participation through participatory research techniques in participatory research. South African Journal of Psychology 39(1): 1–18.

45.

We Can Move (2021) We Can Move – Inspiring People To Get Active – Health and Wellbeing. Available at: https://wecanmove.net/ (accessed 13 July 2023).

46.

Wright

Hayward

West

, et al. (2019) ActEarly: A city collaboratory approach to early promotion of good health and wellbeing. Wellcome Open Research 4: 156.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.03 MB

0.02 MB

0.35 MB

0.01 MB