Abstract
A wide range of evaluative evidence is necessary to capture change within complex systems as findings shape the form and interpretation of outcomes, how they are valued, and actions moving forward. The Atawhai study worked with primary health care professionals in a series of workshops to distil ways to make it easier to respond to family violence as a key determinant of ill health. This article describes three methods used to evaluate the emergence of change from the perspective of the Atawhai research participants including pre/post-readiness surveys, social network analysis and qualitative exit interviews. Each method provides a unique perspective on the complexity involved and triangulation across the methods indicates validation of findings. Capturing change within complex systems requires methodological agility and overlaying of diverse evidence. The contribution of ‘soft infrastructure’ in developing and sustaining complex interventions should not be underestimated.
Keywords
Introduction
Capturing change within complex systems is methodologically challenging due to many diverse and interacting elements that generate uncertainty and unpredictability (Clark, 2013; Moore et al., 2019). How the relationship between the ‘problem’ and the system ‘response’ is conceptualised means it is not always clear what elements are contextual and which are part of an intervention, making assessing outcomes difficult (Callaghan, 2008; Paparini et al., 2021). Controlling for, or obscuring the influence of context, hides part(s) of the complexity involved, making it harder to understand intervention effects across different settings (Moore et al., 2019). Approaches have shifted from addressing the complexity of the problem to recognising how complexity is produced through dynamic interactions between problem, intervention and context (Paparini et al., 2021). A wide range of evaluative evidence is necessary to study impact as findings shape the form and interpretation of outcomes, how they are valued and actions moving forward (Gear et al., 2017; Moore et al., 2019; Paparini et al., 2021). This article considers the use of three evaluative methods: pre/post-readiness surveys, social network analysis and qualitative exit interviews, to explore the emergence of change generated by the ‘Atawhai’ study which aimed to improve primary care responsiveness to family violence in Aotearoa New Zealand. We triangulate findings to infer theoretical implications of the change occurring. This study provides an exemplar of applying complexity-informed evaluative tools within a methodologically congruent research design to capture change occurring within unpredictable and constantly changing complex systems.
Complexity theory and evaluation
While a problem might be conceptualised as complex, scholars do not always make explicit where they theorise the complexity to lie (i.e. context, intervention, problem) or apply methods that align with that complexity (Murdoch et al., 2023; Paparini et al., 2021). Health care systems are now commonly conceptualised as complex adaptive systems (CAS) where patterns of interaction among system elements (inclusive of context, intervention, problem and all actors) lead to self-organising properties and eventually the emergence of new system behaviour (Carroll et al., 2023; Olsson et al., 2020; Thompson et al., 2016). This theoretical positioning has implications for designing and evaluating interventions, requiring critical reflection on the ontological and epistemological assumptions that influence interaction between intervention parts and consequently how outcomes are understood (Clark, 2013; Hawe et al., 2009). Similarly, this lens has implications for research design, necessitating use of multiple and different methodologies and methods responsive to shifts in learning in real time (McDaniel et al., 2009).
Gates et al. (2021) provide a useful summary of five ways which systems and complexity thinking are currently influencing the practice of evaluation, including (1) relabelling existing practices as evaluation of complex interventions, (2) using multiple and diverse theories and methods to evaluate interventions, (3) embodying systems and complexity ways of thinking, being and acting, (4) encouraging innovation, system transformation and ongoing learning and adaptation and (5) transforming the role, environment or field of evaluation. There is also common ground between systems and complexity thinking and Indigenous worldviews, emphasising a relational epistemology where living systems are interconnected (Johnson et al., 2024). In this article, we call attention to how complexity-informed evaluation necessitates a critical reflection on how knowledge has been produced, interpreted and legitimised as ‘evidence’ (Callaghan, 2008; Clark, 2013; Goicolea, 2023; Hawe et al., 2009; Walton, 2016). Within CAS, patterns of local interaction create system behaviour. That is, as people interact, they shape the complex systems they act within, placing boundaries around the nature and value of knowledge produced (Murdoch et al., 2023). Complexity-informed evaluative practice necessitates use of multiple and diverse theories and methods to capture the emergent nature of this socially constructed knowledge and the implications for system outcomes (Murdoch et al., 2023).
The ‘Atawhai’ study
The problem of family violence (FV) is significant in Aotearoa/New Zealand (hereafter referred to as Aotearoa). Population-based data estimates nearly two in three Aotearoa women, over two in three Indigenous Māori women, two in five Pacific women and one in three Asian women will experience a form of physical, sexual, psychological, controlling or economic violence by an intimate partner in their lifetime (Fanslow et al., 2023a). While FV impacts all members of a family, the frequency and severity of violence are heavily directed towards women and children (Fanslow et al., 2023b). The impact of violence on health and wellbeing ranges widely across short- and long-term physical, mental and sexual health problems (World Health Organization, 2002). While international and national intervention models exist (e.g. RADAR, (Harwell et al., 1998) LIVES (World Health Organization, 2014), VIP (Gear et al., 2020)), there is increasing recognition that standard prescriptive interventions do not reflect the complexity of the problem and are unlikely to generate sustainable solutions (Gear et al., 2019; Family Violence Death Review Committee, 2016; Vaughan, 2016). Further, Indigenous-centred approaches to FV are often ‘othered’ or invisible against the dominance of Western bio-medical health care systems, obscuring impacts understood from an Indigenous worldview such as collective intergenerational trauma (Gear et al., 2024b; Pihama and Smith, 2023). In Aotearoa, Te Tiriti o Waitangi is a foundational document that embodies the relationship between the British Crown and Indigenous Māori. Te Tiriti recognises Māori never ceded sovereignty to the Crown and is used to hold the Crown accountable in recognising Māori rights, including addressing Māori health inequities (Waitangi Tribunal, 2023). Table 1 provides an English translation of Māori words.
Māori to English translation.
The authors note that some words can have multiple definitions in the English language, and the translations provided reflect how they are used in the context of this article.
In response to limited health system support, the ‘Atawhai’ (to move cautiously with kindness) study aimed to make it easier for primary care providers in practice to respond to FV as a key determinant of ill health (Gear et al., 2018, 2019). Grounded in Te Ao Māori principles, Atawhai wove together complexity theory and participatory health research methodologies, emphasising the primacy of relationships among those providing care and people experiencing FV. A combined complexity theory and participatory research approach acknowledge no individual can know a whole system, but each person can provide insights into how system parts fit together from their perspective, widening the scope of what is considered ‘evidence’ (Moore et al., 2019; Reed et al., 2018). This methodological lens calls attention to interaction between the continuously shifting inputs, relationships, outcomes and consequences involved in responding to FV in primary health care settings and was applied from research design to analysis of research impact (Gear et al., 2022). Atawhai draws on complexity theorists Cilliers (1998), Morin (2008), McDaniel and Driebe (2001) and Begun et al. (2003) and applies many of the complexity concepts outlined in Braithwaite et al. (2017), providing a pragmatic methodological foundation. It is important to note that Atawhai was designed as a research study, not an intervention.
Conducted in the Bay of Plenty, a region in the North Island of Aotearoa, Atawhai worked with 14 individual primary care professionals in ten 1-day in-person whakawhitiwhiti kōrero wānanga to identify and develop ways to make it easier to respond to FV with primary care settings (Table 2). The fourteen participants included ten females, four males; nine Māori, five tauiwi; two general practitioners, one nurse practitioner, one practice nurse, seven social workers and three health care managers across four general practices (western model of health) and ten Māori hauora (Indigenous model of health). Along with methods to capture change, the primary method of data collection, ‘Kaihopu Kōrero’, involved two senior research team members capturing observation data on how individual and collective participant understanding emerged over the series of wānanga. The Kaihopu Kōrero data set included 18 observation notes. A summary of observations and reflections was provided to the research team and participants following each wānanga (see Gear et al. (2024a) for further study details). The study was approved by the Auckland University of Technology (AUT) Ethics Committee (21/31).
Atawhai timeline.
SNA: Social Network Analysis.
Open to new network members.
Post-research planning (no Kaihopu Kōrero notes).
Atawhai developed a relational response to FV, focused on building quality network relationships to support the delivery of safe, effective and sustainable responses for those accessing primary health care. Participants led the development of the Atahwai Network, a health-sector-led network that connects frontline health care professionals and organisations with information and tools to safely journey with whānau and families experiences of FV and offer opportunities for change (Gear et al., 2024a). A key finding of Atawhai is a lack of critical reflection on the systems and structures that shape how the ‘problem’ of FV is understood (Gear et al., 2024a). During the Atawhai wānanga, participants had the opportunity to critically reflect on the patterns of system interaction that shape the way they practice, initiating transformative change in the way they thought about FV and therefore how they practised (Gear et al., 2024a).
Methods to explore the emergence of change
Use of complexity theory challenges evaluative practice to capture how dynamic interactions between the ‘problem’, intervention and context at local levels leads to system behaviour (Callaghan, 2008). The Atawhai research design (Gear et al., 2022, 2024a) incorporated three methods to capture the emergence of change from the perspectives of the participants: pre/post-readiness surveys, social network analysis, and qualitative exit interviews. The following section describes each method and associated findings.
Method 1. Health system readiness to respond to family violence
There is a wealth of literature on the need for ‘institutional support’ in order to effectively implement responses to FV, particularly intimate partner violence (IPV). Institutional support includes, for example, system infrastructure, guidelines, protocols and evaluation tools (Garcia-Moreno et al., 2015; O’Campo et al., 2011). There is also a large body of literature which explores barriers and facilitators of responding to FV (Gear et al., 2016; Saletti-Cuesta et al., 2018; Tarzia et al., 2021). Yet, evidencing the impact and sustainability of health care responses to FV remains challenging. The concept of ‘readiness’ assesses how ‘ready’, or prepared, a health system may be prior to implementation. Readiness goes beyond typical sensitisation and institutional support to consider, for example, (a) at a systemic level, policy, leadership and community engagement, (b) at a service level, care integration with other services and (c) at an individual level, self-efficacy, motivations, emotions and attitudes (Colombini et al., 2022; Hegarty et al., 2020; Leung et al., 2017; Short et al., 2006).
Atawhai aimed to assess the change in perceived readiness of individual participants, pre and post the series of wānanga. The six items of the online Atawhai Readiness Survey were developed by the research team combining literature on readiness (Barnard et al., 2018; Hegarty et al., 2020; Ministry of Social Development, 2017) across the four domains identified by Leung et al. (2017): self-efficacy, motivational readiness, emotional readiness and workplace readiness. Likert-type scale response options ranged from one (strongly disagree) to five (strongly agree), followed by seven demographic questions. Questions included, for example, ‘I feel confident in clinical skills to support all peoples impacted by IPV’. An online Qualtrics (2020) link to the Atawhai readiness survey was distributed to participants along with introductory information 1 week prior to wānanga one. Participants were invited to complete the Qualtrics survey again following wānanga eight. Data were downloaded from Qualtrics and exported to SPSS for analysis. Given the small sample size, analysis was limited to descriptive statistics of baseline and follow-up data to identify trends.
Thirteen participants completed the pre-wānanga survey. Ranging from one (strongly disagree) to five (strongly agree), mean scores ranged from 3.46 (Self-Efficacy; knowledge and resources) to 4.77 (Motivational Readiness). Eight participants completed the post-wānanga survey. Mean scores ranged from 3.75 (Workplace Readiness) to 4.88 (Motivational Readiness). Between baseline and follow-up, the least change occurred in Workplace Readiness, and the largest change in Self-Efficacy (knowledge and resources). Cronbach’s Alpha demonstrated high covariance among the six survey items (see Table 3).
Participant perceived readiness to identify and respond to intimate partner violence (IPV).
IPV: intimate partner violence.
Notes: 1 = strongly disagree to 5 = strongly agree; Follow-up (completed between 18 to 24 months post baseline); Cronbach’s alpha = 0.85.
Method 2. Social Network Analysis (SNA)
SNA is a set of theories, techniques and tools useful for capturing relationships and interactions between people (Valente et al., 2015). It is increasingly being recognised as a valuable tool in directing the implementation of policy and interventions, calling attention to key individuals, collectives, processes, relationships and communication methods involved (Glegg et al., 2019; Shelton et al., 2018). SNA is often quantitative, with use of common metrics and visualisations (see Hawe et al., 2004 for a helpful glossary of terms), although mixed method approaches are increasing, recognising the importance of qualitative information that contextualises findings (Crossley and Edwards, 2016; Glegg et al., 2019; Luxton and Sbicca, 2021).
In Atawhai, we used SNA to view network structure, how this influences what outcomes emerge and what influences interactions to support the emergence of the desired change. We collected data in a variety of ways including (a) participants themselves adding relationships to their growing personalised SNA map that depicted their connections to others responding to FV, (b) researchers noting relationships with various individuals, collectives and processes during discussion, (c) wānanga photos and (d) participants verbally advising researchers on what relationships to capture. Following each wānanga, we generated a network map of individuals, collectives or processes using Kumu (2021), a data visualisation platform which also generates SNA metrics such as degree, closeness, betweenness and reach. Each wānanga map provided visual formation and evolution of the network over time. Analysis involved visualisations of subgroups, structural holes and opportunities for bridging. External consultation supported metric analysis and interpretation of findings (see Acknowledgements).
Atawhai Network participation grew significantly over the research period (Figures 1 and 2). Post wānanga one, the research team, participants and affiliated organisations were linked, creating the initial wānanga one map with 211 relationships noted among 92 individuals, collectives or processes. Post wānanga ten, 1,026 relationships between 196 members were noted within the Atawhai Network. The number of people a Network member connected with (average degree) at wānanga one was 18.05 (SD 22.80). Due to a considerably expanded network, the average degree at wānanga ten was 13.25 (SD 19.46). On the other hand, the proportion of mutual participant relationships (mutuality) was higher at wānanga ten (0.62) than at wānanga one (0.48), demonstrating higher cohesion at wānanga ten. Overall, there is a high level of mutuality within the Network. However, as more members join (e.g. via the website), there are more who are yet to develop relationships with founding network members, generating a core–periphery structure as shown in Figure 2 (Held et al., 2021).
Network members and relationships across Atawhai workshops.
Atawhai Network growth over time.
Method 3. Participant ‘exit interviews’
Most often used as a management tool to improve organisational performance, exit interviewing can also be used as a research method. Utley-Smith et al. (2006) describe their use of ‘exit interview-consultation’ to extend member checking, validate inferences and conclusions and rapid dissemination of findings. Morrison et al. (2012) describes the importance of an exit strategy to conclude a study ethically and safely, especially when working with vulnerable populations. Of note, exit interviews were not a part of the original research design. Given the limitations of SNA findings, it became evident that exit interviews would provide a reflective process for each participant to review their own intensified network and trajectory of change and provide a reflexive account of the collective journey in which the nonlinear interactions of change may be illustrated.
To capture the story behind the numbers from the readiness survey and SNA data, we collected qualitative data on the impact of participating in Atawhai. The 14 Atawhai participants were invited to participate in an online or in-person semi-structured interview or focus group with a researcher independent of the Atawhai research team. One member of the Atawhai National Advisory Group, tasked with Atawhai scale up and spread, also chose to participate. Informed consent was obtained at the beginning of each interview. Questions aimed to explore participant experience of how research participation impacted them (a) personally, (b) professionally and (c) their network. Interviews were transcribed by the independent researcher and summer student. Given the small sample and close relationships generated by the participatory methodology, participants were advised confidentiality may be limited and they were provided opportunity to review and amend transcripts prior to the Atawhai research team receiving them for analysis. Analysis followed the Atawhai methodology (Te Ao Māori principles and complexity theory lenses) conducted through a ‘Mahi a Rōpū’ (Wilson et al., 2022) approach, capturing working patterns of interaction across a two-day hui with six research team members (three Māori, three tauiwi). Working patterns of interaction were reviewed and agreed on at a follow-up online meeting with four research team members (three tauiwi and one Māori). The whole research team and Atawhai participants were invited to provide feedback on the findings before finalising.
Six Atawhai participants participated in online interviews ranging between 30 and 76 minutes in duration. One Atawhai National Advisory Group member submitted written responses to the survey questions. Analysis generated seven interacting elements of Atawhai that influenced responsiveness to FV (Table 4). Two elements are considered foundational: (1) relationships between tangata whenua and tangata tiriti, and (2) the principles of tika, pono and aroha. Of importance, these foundational elements rely upon both tangata whenua and tangata tiriti committing to practising te Tiriti o Waitangi relationships. We found that as these seven elements interact, reflexive processes occur where knowledge, understanding and change grows for each individual and also collectively, leading to a pathway of deeper understanding about FV responsiveness. This process facilitates exchanges of power between health professionals and care-seekers, promotes Māori leadership of an issue of importance to Māori, and validates evolving practice(s) that are responsive to FV. We suggest this finding is indicative of the complexity involved in FV responsiveness within the context of Atawhai. The pattern is inclusive of context and will change depending on the context in which these elements interact. Table 4 provides the findings of the participant exit interviews, outlining the seven interacting elements influencing responsiveness to FV. Notably, the Table is an abstraction of the data because each element works in concert, exemplifying complexity concepts such as co-evolution, self-organisation, emergence, feedback loops and path dependency (Braithwaite et al., 2017; McDaniel and Driebe, 2001). The research methodology allowed for emergence of new insights into the influence of history, institutional structures as well as growth in individual understanding of FV and FV responses. Participant quotes in Table 4 illustrate how these elements are woven together through interaction, rather than belonging solely to one element.
Atawhai exit interview findings: Elements influencing responsiveness to family violence.
FV: family violence.
Reference frames may include those imposed by an organisational or professional practice worldview.
Discussion
Measuring outcomes from complex systems is difficult, given the number and diversity of elements involved (Paterson et al., 2009). The methodological approach of Atawhai aimed to engage with the complexity, focussing on local patterns of interaction between problem, context and intervention that lead to emergence of responsiveness to FV. This article described the use and findings of three methods designed to capture change emerging from Atawhai. Each of the three methods demonstrated Atawhai was successful in initiating patterns of system interaction leading towards increased FV responsiveness. Although limited by a small sample size, individual readiness survey findings indicated improved readiness to respond to FV (IPV) across all six items, SNA findings demonstrated growth in network size and cohesiveness and exit interviews provided qualitative examples of increased participant confidence, capability and connectedness to respond to FV. Of note, the original research design (Gear et al., 2022) also included clinical practice audit snapshots to assess the rate of IPV identification assessment and referral. Use of this method was abandoned by the research team on the recommendation of research team general practitioners due to anticipated low or non-existent documentation of IPV within the context of limited system support (Gear et al., 2018, 2019).
Use of complexity theory-based evaluation particularly draws attention to the dynamic nature of knowledge production, where different ‘ways of looking’ at the problem and creating sustainable interaction pathways between individuals and institutions are negotiated (Callaghan, 2008). These three methods provided a way to evaluate how change emerged from three different perspectives. Together with Kaihopu Kōrero data and the experience of the research team facilitating the study, Atawhai findings offer a complex picture of the order that was negotiated by participants over time and the effect(s) which emerged (Callaghan, 2008). Triangulating these findings was a reflexive process and we infer two implications for primary care responsiveness to FV.
First, readiness survey findings and exit interview findings both indicate health care collectives (such as organisations and professional bodies) often act as a barrier to change that would support better responsiveness to FV. Readiness survey findings indicate the workplace was considered one of the least ‘ready’ to support a response and changed the least between baseline and follow-up surveys. Exit interviews provided some context to this finding, indicating system structures at different levels are acting as barriers to both study participation, and responsiveness to FV in practice. For example, competing health care priorities, policy directives, political influences, and job protocols and roles prevented participants from utilising learning from, and social capital of, Atawhai. This is an important finding, calling attention to the critical need to shift the collective inertia of health care collectives’ readiness to respond in order to improve potential outcomes for families and whānau seeking care.
Second, findings from all three methods indicate the Atawhai Network fills a significant gap within the health system by bridging clinical and community providers in-service delivery (Burt, 1992). Readiness survey findings indicate the largest increase occurred in participants having ‘the knowledge and resources to identify and respond to IPV across cultures’ (self-efficacy), conflicting with continued low workplace support to respond to FV. This indicates significant social capital developed among participants independent of their workplace that increased confidence and capability in responding to FV. While this was not visible via SNA analysis, this gap was empirically evident from the first wānanga that brought together two often siloed groups in dialogue. The advantage of addressing this gap is captured within ‘perception of the problem’ element of exit interview findings that exemplified the benefit of sharing and learning from others operating from different frames of reference and settings.
A scoping review by Smit et al. (2020) conducted to identify and determine the value of SNA in developing or evaluating complex interventions in health care research found SNA is currently under-utilised in complex intervention development. Atawhai demonstrates an example of applying SNA during the development of a network intervention, capturing the actions and advocacy of network members in ‘preparing the ground’ for responsiveness to emerge (Kavanagh et al., 2022b). In practice, the use of SNA proved to be a static evaluative method at odds with the constant change of complex systems. We struggled to communicate the value and relevance of collecting SNA to participants and consequently the response rate was sporadic and low. SNA data collection relied more heavily on wānanga observation data and the research team noting relationships and organisations involved over time. Constraints of the methods used to collect data prevented further exploring how network structure is influencing what outcomes emerge from the system. Findings were limited to measuring how relationships intensified over time, and opportunities to capture relationships between network members beyond the research participants, and the value (positive or negative) of the relationship were lost. Further, as relationships are in constant change, methods of collection were considered to produce highly subjective data, only ‘true’ at the time of collection. For example, policy changes or employment changes disrupt collegial relationships over time.
The overwhelming evidence of change was located in what has been called the ‘soft infrastructure’ of interventions (Kavanagh et al., 2022a, 2022b). ‘Soft infrastructure’ involves the cognitive, emotional, social and relational resources and processes that support community capacity for change (Kavanagh et al., 2022a, 2022b). Atawhai developed a relational response to FV, recognising the critical need for quality trusted relationships among providers to be able to respond to FV safely and effectively. The resources and tools needed to respond to FV lie within these relationships in the form of intellectual, social and political capital (Lewis, 2010). Lewis (2010) argues that social capital arises from being connected, as well as the purpose for which the relationship is made. Further, the value of social capital passed via established relationships increases as it is afforded additional meaning and legitimacy coming from a trusted source (Kavanagh et al., 2022b). Use of SNA identified a ‘core–periphery’ structure within the Atawhai Network. The ‘core’ is afforded a structural advantage over those network members operating on the ‘periphery’, due to the social capital held within those core relationships. Utilising social capital to connect periphery network members with core members has significant potential for an effective means of scaling up and spreading Atawhai intervention effects (Lewis, 2010; Valente et al., 2015).
Atawhai offers a unique case example of how such a methodological approach can generate naturally occurring interventions that lead towards change. During the wānanga, we saw the ‘magic’ (transformative change) occur as participants grew trusted, quality, interpersonal and interorganisational relationships that enabled them to learn from one another and critically reflect on their own thinking about FV, leading to change in their practice (Gear et al., 2024a). For example, through discussion, participants realised they were not responsible for ‘fixing’ the problem of FV, increasing their confidence in engaging with care-seekers. This builds upon the authors’ previous research theorising the ‘Triple R Pathway’, where a FV ‘Response’ is generated in relationship to the understanding of FV as a health issue (Respond stance), influencing Responsiveness in practice. In Atawhai, providing the time and space for professionals to engage with others from different parts of the system necessitated critical reflection on how the ‘problem’ is conceptualised generating learning which leads to transformative change in practice. While this sounds somewhat simple, what is learnt depends on the nature and quality of the relationships formed (Jordon et al., 2010) and is mediated by many other interacting system parts such as participant job position changes and capacity to attend wānanga, as well as competing management directives, and COVID-19.
Critical reflection and learning also made human-imposed artificial boundaries between the problem, intervention and context visible and re-negotiable. We theorise that the ‘problem’ of FV is not a distinct fixed entity, but is an evolving product of the intervention ‘context’. The intervention ‘context’ is constructed and negotiated by participants as they interact in discussion, and the ‘intervention’ naturally emerges from the interaction between participants (Murdoch et al., 2023). This aligns with Hawe et al.’s (2009) view of interventions as ‘events in systems’ where dynamic system relationships shape intervention effect and impact (Hawe et al., 2009). It is the interactions between people involved that create the reality of the intervention and context, (Murdoch et al., 2023) and, we would argue, the construction of the ‘problem’ itself. It therefore matters who relationships are built between and for what purpose as the bodies of knowledge visible or hidden will have implications for health care equity (Murdoch et al., 2023).
This challenges current guidance for developing and evaluating complex interventions which delineate core elements and phases in complex intervention research (Moore et al., 2019; Skivington et al., 2021). In contrast, Atawhai demonstrates the challenge lies in being agile to the complexity involved, responsive to contextual influences, community- and practice-defined evidence, and inequities continually reinforced by system-held assumptions and biases (Brownson et al., 2022; Moore et al., 2019). In seeking to capture change generated by Atawhai, findings successfully identified system elements that lead towards individual responsiveness to FV as well as elements that disrupt these trajectories. Future research can further explore and develop these elements to continue to shift the system in the right direction. For example, increasing health care organisational readiness to support individual health care providers in responding to whānau and families and addressing system gaps by better connecting clinical and community providers in service provision.
Limitations
This study initiated the development of the Atawhai Network. Findings are based on a small highly localised sample, subject to selection bias. Participants not directly involved in clinical practice (e.g. managers) found it difficult to answer readiness survey items. Despite a full agenda at each wānanga, a specific standing session to collect SNA data would serve to increase response rates. Ideally, data collection would be longitudinal and cross-sectional, (Smit et al., 2020); however, research teams need to assess the cost-benefit of conducting SNA given the high burden in collecting and analysing the data versus the insights provided. The purpose of the research must match the need, and in this study, the qualitative data tell the ‘real story’ behind the numbers.
Conclusion
Atawhai utilised three diverse methods to capture change generated by a study grounded in Te Ao Māori principles, complexity theory and participatory research. Each method provided a unique perspective on the complexity involved and triangulation indicated validation of findings. The Atawhai complexity-informed approach to evaluation focuses attention on capturing change, not assessing outcomes, offering a shift away from identifying linear causal relationships, towards a pragmatic openness to shifting relationships, multiple worldviews and system boundaries (Gates et al., 2021). It necessitates critical reflection on underpinning epistemological and ontological assumptions that set a path dependency towards particular ways of thinking, intervention design and practice outcomes (Gear et al., 2019, 2024b; Clark, 2013; Hawe et al., 2009). As such, complexity-informed evaluation requires the evaluator to take a reflexive and relational approach to research design, conduct and interpretation of data (Callaghan, 2008).
Atawhai critiques how human-imposed distinctions between problem studied, the intervention and context can obscure a view of the ‘natural’ interventions that occur as people share and learn from one another, leading to transformative change. A complexity-informed approach to evaluation provides insight into how parts of the problem, intervention and context are socially constructed, inclusive of power dynamics, political, societal and experiential epistemologies which can mediate outcomes. The contribution of ‘soft infrastructure’ in developing and sustaining complex interventions should not be underestimated. It is the resources and tools within these relationships that add to the social construction of intervention context, shaping how the problem is conceptualised and the idealised intervention. These relationships are key in understanding the emergence of change, as what knowledge is ‘accepted’ or ‘taken up’ is dependent on the decisions and actions of the people involved (Kitson et al., 2018; Salter and Kothari, 2016). The shared relational epistemology of systems, complexity and Indigenous worldviews creates space for different and multiple ways of being and knowing to be privileged in knowledge construction (Johnson et al., 2024).
In alignment with the complexity involved, evaluation intent on capturing change within complex systems should be designed with CAS principles in mind, responsive to change, uncertainty, surprise and unintended consequences (Jordan et al., 2009; Jordon et al., 2010; McDaniel et al., 2009). Complexity-informed evaluation should aim to capture local patterns of interaction leading to system behaviour and interrupt negative patterns that may threaten that trajectory (Callaghan, 2008). Atawhai provides an innovative exemplar of applying complexity-informed evaluative tools within a methodologically congruent research design to capture change occurring within complex systems.
Footnotes
Acknowledgements
Ngā mihi nui ki a koutou te whānau Atawhai. Thank you to all participants, organisations, National Advisory Group and Network members who made this study possible and continue to be committed to the development of Atawhai in Aotearoa/New Zealand. Special thank you to Professor Malcolm Alexander for guiding us in interpreting the social network analysis data and Thea Baillie-Bellew (summer student) for supporting the exit interview data collection process.
Declaration of conflicting interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship and/or publication of this article: Atawhai was funded by the Health Research Council of New Zealand [20/932]. Participant exit interviews were funded by AUT University.
Ethical approval and informed consent statements
This study was approved by the Auckland University of Technology (AUT) Ethics Committee [21/31]. All participants completed written informed consent prior to participating in data collection methods.
