Abstract
Background:
The Systems Evaluation Network (SEN) aims to build capability and capacity regarding the use of systems science in public health evaluation. The SEN was established in June 2021 and 3 years from its inception, we undertook a member survey to understand the engagement with, and impact of, the SEN.
Methods:
An 18-item cross-sectional survey captured quantitative and qualitative responses regarding SEN member perspectives, centring around their experience of the SEN, associated impacts, and future requirements. We analysed quantitative data descriptively and qualitative data through content analysis. Sub-group analyses explored differences between those working in academia vs practice/policy.
Results:
Seventy-three participants completed the survey, with 60% working in academia and 40% in practice/policy. Considering experiences of the SEN, participants felt the SEN has shared information about innovative methods and evaluation approaches (94.0% agreed), has provided the opportunity to share and learn with other members (86.0% agreed), and has improved knowledge of systems evaluation methods (86.2% agreed). Regarding impacts of the SEN, participants stated that the SEN has increased their capability to apply systems-oriented methods and evaluation of systems approaches (76% agreed) and has facilitated relationships with others (56.9% agreed). Participants shared future capability requirements for evaluation, which focused on methods (e.g. systems dynamics modelling and ripple effects mapping), approaches (e.g. developmental evaluation and embedded researchers), and other ways in which capability could be increased (e.g. by using case studies).
Conclusion:
This paper illustrates the experiences and impacts of the SEN, identifying its strengths such as the wide range of topics/content and the flexible and accessible delivery format, but contrast against the difficulties of fostering new relationships in an online setting. These findings can help inform the future direction of the SEN and provide insight to other online communities of practice.
Introduction
There has been a growing interest in, and adoption of, systems science within the public and population health arena in the last decade.1–7 There is now widespread recognition that many non-communicable diseases and planetary challenges (e.g. climate change) emerge from complex adaptive systems;8–10 systems with many factors at play, all highly interconnected, which evolve and adapt over time in largely unpredictable ways, and lead to outcomes that are greater than the sum of their parts. Moreover, beyond recognising that many contemporary issues have arisen from complex adaptive systems, the intervention literature advocates for the adoption of systems approaches.8,10–15 Systems approaches respond to the complexity of the challenge, galvanise support from multiple organisations, sectors, and communities, and through co-ordinated action, and seek to fundamentally change how the system functions (i.e. that which is causing the issue to occur).
In 2021, Jebb and colleagues – through the UK Academy of Medical Sciences and the Canadian Academy of Health Sciences – summarised current progress regarding systems-based approaches in public health. 9 Their conclusions, echoed by other reviews,1,2,5 highlight a large variation in how systems science is being applied. This collective research also concludes that academic jargon and concepts are making systems-based approaches challenging for practitioners to apply in the real world. In academia, there is also limited technical expertise for how to utilise systems science in practice given the interdisciplinary nature of the field. Thus, to advance the current state of play, Jebb and colleagues 9 conclude that communities of practice are required to connect interdisciplinary researchers, practitioners, and policy-makers. They also call for methodological advancement to better understand complex adaptive systems and the approaches for intervening in such systems.
In response to these calls, and based on our collective experience, we launched the Systems Evaluation Network (SEN) in June 2021. The SEN is a free, online, international network which aims to: (1) improve evaluations of systems approaches in public health; (2) promote others to adopt, refine, and value alternative research designs and methods in the field; and (3) work with research councils and other funding agencies to generate more opportunities for evaluation of systems approaches to be carried out. We aim to do this by bringing people together who are involved in evaluating systems approaches, to exchange knowledge and provide upskilling opportunities through network meetings and sharing of resources, and by promoting collaboration between SEN members.
The SEN meets online every 3–4 months for 2 hours, with topics, content and format tailored to the preferences of network members. The membership has grown substantially from approximately 100 members in June 2021 to 780 members as of September 2024. Of these, 49% are from academia, 46% from practice/policy, and 5% from independent organisations. Most SEN members are based in the United Kingdom (79%), with smaller groups in Australia (3.5%), Denmark (3.2%), the Netherlands (3.0%), the United States of America (1.9%) and Canada (1.3%). To date, the SEN sessions (n = 12) have focused on specific methods (e.g. social network analysis, systems dynamics modelling, agent-based modelling, ripple effects mapping), the conceptualisation of systems evaluations (e.g. developmental evaluation), and illustrative and innovative case study evaluations. Meeting registrations have ranged from 100 to 242 people (see Table 1) with attendance between 50% and 60% total sign-ups (data not routinely collected). SEN members also have access to an online repository of resources (e.g. journal articles, method guides), a YouTube channel for previous session recordings, and a SEN member list to promote collaboration.
Overview of SEN sessions
Given that the SEN has been operational for 3 years, in this cross-sectional study we aimed to (1) understand SEN members’ experiences of the network; (2) understand the wider impacts of the network; and (3) determine future capability building requirements to advance the evaluation of systems approaches in public health.
Methods
Study design
We designed an 18-item cross-sectional survey, orientated around the aforementioned research aims, to share with members of the SEN. The survey had four main sections: (1) About you – questions to ascertain information about each survey respondent, including their professional background, expertise, and reasons for joining the SEN; (2) Engagement and impact of the SEN – questions to understand SEN session attendance, and whether the SEN was meeting its intended aims; (3) Future focus – questions to explore what the future capability needs are for SEN members; and (4) Feedback on the SEN – questions to delineate what is working well within the SEN and where there are opportunities for improvement. Questions varied from seven-point Likert-type scale responses to open-ended responses. See supplementary material for survey. Ethical approval for the study was provided by the Leeds Beckett University Research Ethics Committee (ref: 118338).
Participants
We recruited participants via a network wide email in November 2023 (n = 735 members). A subsequent follow-up email was sent two weeks later to remind members of their invitation to participate, and additional prompts were shared via the SEN X account. A link to a Qualtrics survey was included in both emails. Participants were reminded that their participation was voluntary and that complete anonymity was ensured. Informed consent was required before accessing survey questions. The survey was open between November 2023 and January 2024.
Data analysis
The quantitative data from the seven-point Likert-type questions (e.g. to what extent has the SEN increased your capability to apply systems-oriented methods or evaluate systems approaches?”) were analysed descriptively in SPSS. Sub-group analyses were performed to examine differences between those working in academia, as opposed to practice/policy. The decision to make this distinction was based upon the main stakeholder groups that the SEN aims to engage, and which data is routinely captured upon (e.g. when individuals sign up to an SEN session, they are asked to state whether they work in academia or practice/policy). Pairwise deletion maximised the data available for analysis. The data from the open-ended questions (e.g. what methods, approaches, or training needs should the SEN focus on in the future?”) were inductively analysed using content analysis 16 by JN, with a second researcher (AP) double coding a sub-set of responses, and discussion between the two to agree on final themes. Quotes from participants are used to convey some important themes. Results are organised to answer the research aims, bringing together the quantitative and qualitative findings.
Results
Survey response
A total of 73 individuals (~10% of all members) participated in the survey. Of the 63 that provided information about themselves, 60% worked in academia and 40% in practice/policy. Six of those working in academia also had an embedded role within a local authority/heath system. The respondents had methodological and/or research expertise (e.g. systems science, behavioural science, social science), topic expertise (e.g. physical activity, obesity prevention, mental health), applied public health and/or sectoral expertise, or management or leadership expertise. Their reasons for joining the SEN broadly fell into four groupings: (1) an interest in the use of systems science and/or thinking; (2) an interest in systems-orientated evaluations; (3) wanting to access peer support; and (4) to learn from others. Table 2 presents the results from the quantitative data collected, and the narrative relating to each theme combines the quantitative results with the qualitative themes.
Quantitative perceptions regarding the SEN’s utility
Experiences of the SEN
94% agreed that the SEN had shared information about innovative methods and evaluation approaches. Those working in practice/policy were more likely to strongly agree than their academic counterparts (47.6% vs 37.9%). From the qualitative responses, participants (n = 25) reflected on the quality of the presentations, content, and the speakers: ‘variety of the speakers, communication of events, availability of slides and presentations, overall content is varied’ [academia and practice/policy], ‘the content and topics covered at each session are consistently of interest’ [practice/policy], ‘the size and breadth of the network helps me to understand the scope of the work in the UK and beyond. It’s great to have perspectives and experiences from beyond academia’ [academia], and ‘I really enjoy the real-life examples presented in the sessions’ [academia].
86% agreed that the SEN has provided the opportunity to share and learn with other members. Academic members were more likely to strongly agree than those from practice/policy (48.3% vs 33.3%). From the qualitative responses, six participants shared insight about their experiences relating to sharing of resources: ‘the content, organisation, and sharing of resources is all excellent’ [academia].
86% agreed that the SEN has improved their knowledge of systems-oriented methods/evaluations of systems approaches. Those in practice/policy were more likely to strongly agree than those in academia (28.6% vs 20.0%). This was also demonstrated in the qualitative comments, where participants (n = 12) reflected on the relevance of the SEN content in improving knowledge: ‘SEN is developing a good network of individuals, which provides opportunities for collaborations and access to subject expertise. The content of the sessions to date has seemed very appropriate for the systems evaluation focus of the group’ [academia].
Impacts of the SEN
76% agreed that the SEN has increased their capability to apply systems-oriented methods/evaluations of systems approaches. None of the practice/policy members strongly agreed with this statement, compared to 20.7% of academic members. From the qualitative responses, participants (n = 6) shared how the SEN had increased their confidence in using or advocating new methods: ‘It [the SEN] has influenced my understanding and ability to articulate complex systems approaches and their evaluation. I feel confident to champion complex evaluation approaches in my current role’ [practice/policy]. Participants (n = 13) also shared how their increased capability and confidence led to new ideas being developed: ‘Being connected and participating in these sessions keeps me in touch with new methods and gives me new ideas’ [academia], ‘identified areas that I need to develop in order to conduct my own systems and social network analysis research. It has sparked ideas for my research’ [academia and practice/policy], and ‘expanded my toolkit and horizons in terms of what can be done in terms of systems evaluation and who to contact for support’ [academia].
57% agreed that the SEN has facilitated or developed their relationships with others. Here, practice/policy members were more likely to disagree with this statement than academic members (28.6% vs 3.3%). From the qualitative responses, participants (n = 17) spoke specifically around how the development of relationships has led to collaborations: ‘I have developed a collaboration through connections within the SEN which has led to the development of an ongoing research project. This has not involved applying for funding, but the intention is to co-produce a publication from this work’ [academia] and ‘it [the SEN] has helped me to consider the wider “world” and examples of systems approaches, outside of my locality. I’ve made connections with other SEN researchers which has led to collaborative work’ [academia]. Developing such relationships also led to participants (n = 3) feeling less isolated in their work: ‘Gaining inspiration from the meetings and feeling less alone in this field’ [academia], ‘Feeling part of a community is a big plus!’ [academia and practice/policy], and ‘it’s been so valuable over the last couple of years and helped me to feel less lonely. I’ve learnt a lot and benefitted from the webinars and connections with other SEN members’ [academia and practice/policy].
Future capability requirements for evaluation
Participants shared future capability requirements and specified methods and/or approaches they feel they need further support on (some of which have already been the focus of subsequent SEN sessions). These included specific evaluation methods (n = 17), such as systems dynamics modelling, soft systems methods, viable system model, and contribution analysis. Participants (n = 4) also shared specific approaches to evaluation they would like to learn more about, such as developmental evaluation, embedded research, and community participation within evaluation. Developing capability around theory was also something participants shared (n = 4), particularly to help understand unintended consequences and identify leverage points. Drawing on case studies to demonstrate the use of these methods and approaches was something participants (n = 4) wanted to see more of. Two participants felt it was important to develop a shared understanding of systems to facilitate increasing the capability and usage of systems evaluation approaches, such as having a common language of systems and identifying the value and limitations of systems evaluations.
Discussion
Within systems-based approaches in public health, communities of practice – such as the SEN – are urgently required to connect interdisciplinary stakeholders. 9 It is therefore imperative to share the experiences of such communities which are seeking to work in this space. This paper – which links to an active and international network of over 750 academics, practitioners, and policy-makers – presents valuable information about members’ perceptions and experiences of the information shared, impact, and future capacity building requirements of the SEN.
Experiences of knowledge sharing through the SEN
The results indicate a clear agreement that the SEN has enabled information sharing and learning, and improved knowledge of systems-orientated methods and evaluation of systems approaches. Interestingly, those within practice/policy were more likely to strongly agree with these points, highlighting the strength of the SEN to enable knowledge translation and sharing between sectors. In addition, almost half of the whole network consists of stakeholders situated outside of academia, further demonstrating the potential scale of the SEN to promote the uptake of systems science in practice and policy. This helps to overcome the challenges reported within systems-based approaches, whereby academic literature and terminology can be challenging to navigate, leading to a perception that systems-based approaches are difficult to apply within public health. 9
Qualitative responses from participants help explain why the SEN has been successful with information sharing and knowledge translation. First, variety in the information shared (i.e. topics and content – see Table 1), throughout the SEN speaks to a wide audience. This is helped by the size and diversity of the network, who contribute to the content delivered, and feed into the agenda setting for future sessions. Furthermore, participants noted the benefits of sharing real-life examples, demonstrating how systems science can be applied in different contexts. The application to real world contexts could explain why the SEN has helped to overcome some of the challenges associated with understanding systems-based approaches, especially for those in practice/policy. In doing so, presenters draw on their tacit knowledge from applied experience that may not be captured and disseminated in scientific publications.
Second, participants noted the flexibility in how information is shared throughout the SEN as another strength. Members of the SEN are given the option to attend live online sessions or to access the recorded content shortly afterwards. Evidence reports the benefits of learning through online platforms, highlighting that the combination of live webinars and archived recordings increases accessibility while remaining educationally acceptable among trainees in low- and high-income settings. 17 A further strength of virtual communities of practice is that they provide individuals with the opportunity to engage, which would otherwise not be possible due to time limitations or geographic barriers, including time-zone differences. 18 When participants were asked about the future capabilities of the SEN, maintaining a wide scope of methods and case study examples related to the evaluation of systems approaches were highlighted as important. This combination of topics that appeal to a wide range of stakeholders, alongside flexible and accessible content delivery, has enabled the development of a ‘connected global community’ with those interested in systems-based approaches in public health. 9
Perceived impact of SEN
Overall, participants felt that the SEN had increased their capability to apply systems science methods, but we noted interesting differences between groups. While the knowledge of practice/policy stakeholders was more likely to have increased (see above), here, academics were more likely to report capability increases. This is not however surprising. It seems reasonable that practitioners and policy-makers may need to be aware of systems science methods (for example, understanding what they are, what they do, what they can learn from them), but application of these methods are still perceived as more closely aligned with the skill set of academics. 19 Furthermore, we would not expect the SEN alone to sufficiently increase capability to apply systems-orientated methods; it introduces people to such methods and signposts to appropriate resources. We also hope that the SEN provides a platform to help diverse stakeholders to build an understanding of what approaches exist, including for what purpose, and to see value in collaborating with those with expertise in using systems-orientated methods.
However, when asked if the SEN had facilitated or helped to develop relationships, only half of the respondents agreed. While some respondents provided positive experiences of relationship building through the SEN, this was not something experienced by most participants. This is likely due to the online nature and presence of how the SEN currently operates. Although, the online nature can reduce specific logistical barriers, 18 it has been reported that developing trust can be difficult in virtual communities of practice, which can then have a significant impact on the development of professional relationships.20,21 Current evidence also highlights digital communications, such as video calls, cannot replace in-person meetings 22 and that developing networks and building future collaborations is still a benefit of in-person events. 23 While the SEN promotes collaboration between its members through the sharing of members work and contact information, further work may be required if relationship development is a central tenet. Offering further communication methods (e.g. discussion forums) or opportunities to meet in person may help the development of future professional relationships. 20 Networks such as the SEN need to consider how to feasibly enable such methods, while also accounting for the challenges within volunteer led work (e.g. technical resource, personnel resource). One suggestion was to align with already existing organisations that have the capacity to support in-person events. For the SEN, this facilitated the development of the Systems Science Special Interest Group in 2024 through the International Society of Behavioural Nutrition and Physical Activity.
Strengths and limitations
The collection of quantitative and qualitative data from both those within and outside of academia, allowing for a deepened understanding of the experiences and impact of an active online community of practice (i.e. SEN) can be seen as a key strength of the current study. However, limitations with this work also need to be noted. First, although there was diversity in respondents there was an overrepresentation of academics (60% vs 49%), and an underrepresentation of practice/policy (40% vs 46%) and independent organisations (0% vs 5%) when compared to the total SEN membership. Gaining a greater number of responses and closer representation to the total SEN membership may have influenced the findings of the study, especially the perceived experiences from qualitative responses. Second, while the use of open responses in the survey allowed for the collection of qualitative data, which helped to obtain responses from a greater number of participants, this approach does not allow for the prompting of further information. Interviews or focus groups would have enabled prompting of participants; which could be a suggestion to further assess the experiences and impact of online communities of practice within members. Third, it is important to acknowledge that the survey received a 10% response rate, which may mean that our sample – and our findings – do not represent the perspectives of the remaining 90%, likely leading to more favourable responses. And finally, given the authors roles in organisation the SEN, we must be mindful about the potential for bias in the analysis and reporting of findings. We have ensured that this evaluation is transparent and comprehensively reported, and used double coding for the content analysis. This evaluation collectively provides a critical reflection on both the strengths and limitations of the SEN.
Conclusion
The current study aimed to assess and share the experiences and impact of an active online community of practice, the Systems Evaluation Network. Through the sharing of real-life examples which drawn on applied tacit knowledge, a combination of topics that appealed to a wide range of stakeholders, and a flexible and accessible delivery format, the SEN has enabled information sharing and learning, and improved members knowledge of, and capability to apply, systems-based methods and approaches to evaluation. A notable challenge was the ability of the online format to foster professional relationships, and consideration needs to be given to the feasibility of complementary communication methods and/or events that would more readily allow such collaborations to develop. These findings can help inform the future direction of the SEN, while also aiding the success of other online communities of practice.
Supplemental Material
sj-docx-1-rsh-10.1177_17579139251403525 – Supplemental material for The systems evaluation network: building capability and capacity in the use of systems science across public health
Supplemental material, sj-docx-1-rsh-10.1177_17579139251403525 for The systems evaluation network: building capability and capacity in the use of systems science across public health by J Nobles, J Murphy, D Radley and A Potts in Perspectives in Public Health
Footnotes
Acknowledgements
We would like to thank the organising committee for their continued support with the SEN and in designing and developing the survey: Dr Yanaina Chavez-Ugalde, Dr Hannah Forde, Dr Leandro Garcia, Professor Valerie Gladwell, Dr Jennifer Hall, Dr Mohammad Hassannezhad, Dr Kelly Mackenzie, Dr Katie Shearn, Naimh Smith, Professor Nick Cavill, Dr Erica Gadsby, Dr Dylan Power, Chris Perks, Dr Alexia Sawyer, Dr Felix Gradinger, and Dr Thomas Mills. We would also like to thank all those who have engaged with the SEN thus far, both members and presenters.
Conflict Of Interest
The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: The authors of this paper are part of the SEN, with James Nobles and Alexandra Potts being co-founders. There are no other competing interests to declare or associated funding.
Ethical Approval
Ethical approval for the study was provided by the Leeds Beckett University Research Ethics Committee (ref: 118338).
Supplemental Material
Supplemental material for this article is available online.
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
