Developing an M&E system for a wicked problem

Abstract

Wicked problems are a challenge due to their dynamic and uncertain behaviour, inter-relationships with multiple systems and sub-systems, actors or non-human variables, and scale. Monitoring and evaluation (M&E) is a crucial element of any intervention seeking to contribute to positive change. This article explores a process used to design an M&E system that responds to a wicked problem. The authors argue that wicked problems require an M&E system design that is grounded in systemic thinking. Such an approach has been adopted to design an M&E system for counter-trafficking in persons by the Australian Government funded Australia and South East Asian Nations (ASEAN)-Australian Counter Trafficking program. This article describes the process used to develop the M&E system and the value of using systemic thinking to inform M&E system design for wicked problems and concludes with lessons learned.

What we already know

• Wicked problems are particularly challenging and increasing in complexity with the rise of polycrisis, or multiple wicked problems intersecting.

• The challenge extends to the question of evaluation methodology. We know that reductionist paradigms are not ‘fit for purpose’.

• Trafficking in persons, is a global wicked problem that is being addressed in the ASEAN region with support from the Australian Department of Foreign Affairs and Trade.

• ASEAN is eager to establish monitoring and evaluation practices to measure and report on counter trafficking practices. However, there are a number of barriers to the adoption of M&E, including low levels of M&E literacy and reporting practices.

The original contribution the article makes to theory and/or practice

• Through this applied work, using a post-positivist evaluation approach (the Inclusive Systemic Evaluation for Gender Equality, Environments and Marginalised Voices), we are able to take the principles of systemic evaluation and apply these to develop an M&E System for the ten diverse ASEAN member states. This article is unique in that it describes what we did, how we did it, and the lessons we learned about using systemic evaluation principles for the design work of an M&E system for wicked problems.

What is a wicked problem and why do they need M&E?

Complex societal challenges that are associated with significant negative impacts for humanity, and our environment, are often called ‘wicked problems’ (Crowley & Head, 2017; Grewatsch et al., 2023; Rittel & Webber, 1973). Wicked problems occur across many aspects of society, locally, nationally, and globally and are ‘characterised by multiple stakeholders involved in complex and unpredictable interactions’ (Williams & van’t Hof, 2014, p. 2). Wicked problems, as complex systems, may hold an almost limitless number of constantly changing relationships and interactions between actors and systems. Uncertainty often arises from feedback loops and a system’s characteristic known as ‘emergence’, where a new phenomenon arises through interactions in the system, but which may not resemble the system or subsystems of its origin (Ison, 2008). Wicked problems are interconnected through a complex series of dynamics (Crowley & Head, 2017; Grewatsch et al., 2023) and have the potential to escalate into a polycrisis (Morin et al., 1999) where systems become causally entangled in ways that significantly degrade humanity’s prospects. For example, natural disasters, poverty, and conflict situations are associated with increased rates of trafficking in persons (International Labour Organization, 2022). Understanding and solving wicked problems requires reliable data, however, wicked problems are often plagued by ‘data scarcity and high dimensionality’ (Lavelle-Hill et al., 2021, p. 1). The factors influencing data scarcity about wicked problems are complex. These may include geographic, resource, cultural, digital, economic, and geo-political challenges all of which must be considered.

Why a systemic approach is required for working on wicked problems

Wicked problems may be viewed as a series of interacting systems, some changing slowly and others moving more dynamically (Grewatsch et al., 2023). Wicked problems can be understood using systemic thinking because it takes into account the complexity of interactions and ongoing change with an array of ideas and techniques (Gates, 2016) for an almost limitless number of variables (Williams & van’t Hof, 2014). Systemic thinking can be described by comparing it to systematic thinking. Systematic thinking is an objective and results focussed approach which is orientated to a linear progression of tasks. Systemic thinking is different because it regards a complex system as more than the sum of its parts and studies that system holistically. Evaluation practitioners may be systematic at times, but it is systemic practice that is reflexive, responsive to change and emergence, and thereby able to accommodate the diversity of interests and perspectives embedded in systems (Ison, 2008; Reynolds et al., 2016).

When thinking about the evaluation of wicked problems, at least three elements need to be considered. These include the system’s boundaries, as these determine what is ‘in’ and what is ‘out’ of a system and provide some comprehension as to the nature of the system (and the problem), an understanding of the inter-relationships and connections between people, systems or ideas, and the actors associated with the issues of interest. The various perspectives of key actors who hold differing viewpoints in relation to their position within the system/s are important to understand (i.e. decision-maker, policy analyst or advisor, practitioner, and beneficiaries) (Gates, 2016; Midgley & Pinzon, 2011; Stephens et al., 2018; Williams & Hummelbrunner, 2010; Williams & van’t Hof, 2014). Taken together inter-relationships, perspectives, and boundaries are ‘the essence’ of systemic thinking according to Reynolds et al. (2016) and ‘distinguish a systems approach from other ways for dealing with complexity’ (Reynolds et al., 2016, p. 667).

Reflection on the interrelationships, perspectives, and boundaries of the evaluand and evaluation help us to manage our approach and execution of an evaluation project. It is important to note that interrelationships, perspectives, and boundaries are interlinked. For example, thinking through the range of perspectives that may be relevant to the wicked problem in focus, may inform a stakeholder analysis, how the evaluation is framed and conducted, method selection, the validation of an evaluation’s findings, and its end-use (Gates, 2016). Reflection upon these may then lead to a widening of the boundary of the evaluation to consider and include new actors (Stephens et al., 2018). Reflecting on the actors’ perspectives and the interrelationships between people, systems, and ideas, evaluators are challenged to think beyond the funder or donor’s view of a program, policy, or project, or that of the project and program staff, to consider the power dynamics with consideration of the marginalised voices at play, and how the end-user is able to take up and apply the end-product, all of which we showcase in the design of an M&E system, the subject of this article outlined below.

An M&E system sits behind applied evaluation practice and consists of component parts most evaluation practitioners will be familiar with such as a logic model, theory of change, evaluation framework, guidance, and tools. When designing an M&E system it may be tempting to take a reductionist view and define the components and outcomes of the system in a systematic way based on a few significant variables to design the ‘right’ solution or eliminate the problem altogether (Gates, 2016; Reynolds et al., 2016). However, for wicked problems that do not have singular causes, effects, or solutions, focussing on just a few significant variables is unlikely to increase understanding or insight. Given that wicked problems occur in a social-ecological context, adopting a systemic (or systems) thinking lens may be more helpful (Grewatsch et al., 2023). As Gates (2016) has stated: ‘Social problem solving becomes an ongoing, iterative process of learning about social problems and adaptive management’ (p. 66).

Consideration of the knowledge generated and end-user application makes implementation and change management an important part of the design process of an M&E system. Effective implementation and change management need to be systemic and designed for systems that are living and evolving. A systemic approach to change management promotes adoption and evolution through the interconnections between the diverse actors, ideas, systems, and sub-systems (Cao & McHugh, 2005; Vlados et al., 2018).

This article describes an approach to the design of an M&E system for the wicked problem of human trafficking. We propose that the systemic approach to the development of an M&E system taken here could be applied to other socio-ecological wicked problems, in the same way that the evaluation of a wicked problem should be systemic. As the precursor to evaluation practice, the M&E system is informed by systemic thinking principles. This article describes the component parts of a system designed for the evaluation of human trafficking across South East Asia (the ASEAN group of nations) and the processes we used in the design. Where possible we highlight the differences of a systemic approach and value for the end-user to support their systemic evaluation literacy and capacity. The project was commissioned through ASEAN-Australian Counter Trafficking in partnership with the ASEAN Commission on the Promotion and Protection of the Rights of Women and Children (hereon ‘the Commission’) in 2023.

Inclusive systemic thinking and the gender equality, environments, and marginalised voices framework

The Inclusive Systemic Evaluation for Gender Equality, Environments, and Marginalised Voices (Lewis e al., 2024; Stephens et al., 2018) was selected as an overarching systemic evaluation approach to support the design of the M&E system and project implementation. This approach guides evaluation practitioners to make judicious considerations and selections of systems boundaries, and to look for the intersections and interrelationships these may have with other systems’ boundaries.

All actors carry with them perspectives that stem from their present and historical positions, values, beliefs, and roles in the systems. Their identity is intimately tied to how the systems are seen. This then places the evaluation practitioner as part of the system. Practitioners need to critically reflect on the perspectives of the agents operating within these systems as well as their own knowledge, identity, and relationships with others (Midgley, 2023). Evaluators’ practice requires regular and critical self-reflection of biases that can arise from one’s positionality (Gregory, 1996; Ulrich, 1994) and the evaluator practitioner is no exception to this requirement.

The Inclusive Systemic Evaluation for Gender Equality, Environments, and Marginalised Voices was used in several ways. Part A of the Guidance was used to conduct systems thinking capacity development with the project evaluation team. This ensured a common language about systems thinking, awareness of the systemic thinking paradigm and why the project plan provided to ASEAN reflected a developmental cycle for continuous learning and improvement.

Part B of the Guidance document provides a comprehensive approach to planning including deliberative reflection on the interrelationships, perspectives, and boundaries of the evaluand and evaluation. This included the scoping of boundaries for context and problem structuring; selection of components for the M&E system (see below) and the application of the Gender, Environments, and Marginalised Voices or ‘GEMs’ framework, to the consultation process. The Gender, Environments, and Marginalised Voices framework is a mode of analysis to promote fairness, equity, and inclusivity across organizational policies, programs, and initiatives.

The Gender, Environments, and Marginalised Voices framework is used in evaluation practice to reflect on the relevant elements within the social and ecological dimensions pertaining to gender, environmental systems, and marginalisation. Each dimension acts like a lens to work in close and intentional partnership with local actors who are capable of naming who or what is impacted, along with who and what should be excluded or included, and why. This process of systemic boundary critique identifies many diverse perspectives and promotes the interests of oppressed groups who are disproportionately affected by many of the world’s intractable wicked problems and should be included in M&E activities.

What is trafficking in persons?

Trafficking in persons is a complex, multi-systems, large scale problem (Rittel & Webber, 1973). Also known as human trafficking or modern slavery, it involves recruiting, transporting, harbouring, or receiving individuals through force, coercion, abduction, fraud, deception, or abuse of power (ASEAN, 2016). It encompasses various forms of exploitation such as prostitution, forced labour, slavery, and child exploitation, affecting up to an estimated 50 million individuals worldwide (International Labour Organization, 2022).

Significant strides have been made in global counter trafficking in persons efforts since 2000. Governments, international organizations, and non-government organisations have intensified their efforts, focussing on prevention, investigation, prosecution, and protection measures. There has been a marked increase in international collaboration, highlighted by national and international initiatives (Heintze & Lülf, 2016; McGaha & Theiss, 2007). Additionally The Convention Against Trafficking in Persons Especially Women and Children (hereon ‘the Convention’) was ratified by ASEAN in 2015 with a focus on awareness campaigns, victim support services, and enhanced law enforcement training.

The Commission has developed purposive guidelines to aid ASEAN member states and related stakeholders to implement the Convention. These guidelines represent a substantial improvement in policy and frontline practices over the last 20 years. In Southeast Asia, the ASEAN-Australian Counter Trafficking partnership has strengthened prevention, response, and justice sectors within ASEAN member states. The partnership aims to align efforts with the ASEAN Convention by enhancing frontline responders’, ‘who are directly involved in the assistance and protection of victims of trafficking’ (ASEAN Commission on the Promotion and Protection of the Rights of Women and Children, 2022, p. 1), capacity and developing guidelines informed by victim-survivors’ experiences. Frontline responders, and the end-users of this work, includes non-governmental organisations, practitioners and law enforcers, prosecutors, judges, and policy makers (ASEAN Commission on the Promotion and Protection of the Rights of Women and Children, 2022). Frontline responders work in community settings, such as shelters, criminal justice, and border protection agencies, and in government. There are challenges in providing coherent coordination and communication across so many agencies, including data scarcity and inconsistent data collection prompting recommendations to improve data collection and analysis (Sumner, 2020). A unified understanding, dialogue, and approach to organisational and regional data collection is needed for analysis and reporting and improved focus on victim centred and gender sensitive approaches to counter human trafficking. One way to achieve this has been to develop an M&E system for the Commission about the extent of the adoption and implementation of the Commission’s counter trafficking guidelines.

Introducing the project

In 2022, the Commission tendered for the development of an M&E system on the adoption and monitoring of gender-sensitive and victim-centred approaches to counter trafficking in persons. Three pivotal guidance documents aligned to the remit include The Regional Guidelines and Procedures to Address the Needs of Victims of Trafficking in Persons (ASEAN Commission on the Promotion and Protection of the Rights of Women and Children, 2018); Gender Sensitive Guideline for Handling Women Victims of Trafficking in Persons (ASEAN, 2016), and The ASEAN Do No Harm Guide (ASEAN Commission on the Promotion and Protection of the Rights of Women and Children, 2022). The authors, as awardees of the contract, designed an M&E system to help evaluators and frontline responders achieve a common understanding, consistent approach and quality monitoring and reporting on the adoption and implementation of the Commission’s guidance for counter trafficking, and that are gender-sensitivity, child-friendly, and victim-centred.

A key requirement of the M&E system was to assess the adoption and implementation of victim-centred and gender-sensitive approaches in addressing trafficking in persons, as well as to encourage the use of established guidelines. Given the complexity and sensitivity of this subject, and the involvement of numerous diverse stakeholders, extensive consultation was essential. To do this, several integrated components of the M&E system were designed. Each component was designed and tested to ensure it is easy to use and accessible, and presented in the format and language of M&E. All components come together in one simple guidebook in an interactive PDF format (Guidebook for Monitoring Gender Sensitive and Victim Centred Approaches to Countering Trafficking in Persons, hereon ‘the Guidebook’) (ASEAN Commission on the Promotion and Protection of the Rights of Women and Children, 2024). The Guidebook’s implementation is currently in pilot reporting mid-2025. Pilots are testing how the Guidebook facilitates reflective practice and learning, where strengths and opportunities for capacity development and system change are highlighted, and systemic thinking and application that supports frontline workers’ counter trafficking in persons efforts.

The next section describes each component of the system, followed by key learnings and discussion regarding the extent to which we approached this task as systems thinkers and the design of an M&E system for counter human trafficking work going forward.

The M&E system

The components of the M&E system include: the M&E Framework and underpinning logic model, an M&E self-assessment tool, named the Building Block Maturity Framework for each immediate outcome, and a supplementary Maturity Model Outcome Tracking Tool. The interactive PDF Guidebook became the face of the M&E system. Figure 1 illustrates the M&E system and Table 1 details each component.

Figure 1.

The M&E system.

Figure 2.

‘How to use this guidebook’ page 17.

Figure 3.

Building block and maturity framework.

Table 1.

The M&E components.

The M&E Framework	The M&E Framework serves as the primary structure for the evaluation system. It is underpinned by a comprehensive logic model, developed with inputs from the three guidance documents, relevant literature, and extensive consultation (see below). The framework is organized around six key areas of activity (see Table 2) related to countering trafficking in persons that served to organise the sequence of the outcomes in the Guidebook. It includes 20 indicators aligned with these key areas and leads to the formulation of 40 immediate, 29 intermediate, and 26 long-term outcomes. The outcomes direct users to consider new or amended policy, the tightening of operational procedures, coordination between agencies, and communications. Each outcome represents a body of knowledge and practice, some intersect across more than one other key area, while others are stand alone
The M&E system ‘guidebook’	The guidebook encompasses all components of the M&E system and provides definitions, terms, and directions for its use. The subject of each page is an immediate outcome, accompanied by detailed guidance notes with footnotes to source documents and a monitoring checklist for each immediate outcome (see Figure 2). Intermediate and long-term outcomes are presented on each page to provide the reader a sense of ‘where we are heading’. It promotes the adoption and implementation of the Commission’s guidelines, reaching and supporting all segments of society, especially vulnerable groups such as women, children and persons with disabilities
Self-assessment and the Building Block Maturity Framework	The Building Block Maturity Framework (see Figure 3) allows practitioners to conduct self-assessment of their achievement of outcomes at various levels. It accommodates the variation in capacity and capability among ASEAN member states and includes a simple rubric for self-assessment of performance levels. It is designed for evaluation practitioners and frontline responders to engage easily with reflective practice, and with the interactive checklist provided, monitor the progress they are making
Maturity Model Outcome Tracking Tool	The Maturity Model Outcome Tracking Tool was developed as a prototype using MS Excel to monitor progress in adopting and implementing the key guidelines at the organisational, regional or national level. It was delivered as a supplement to the Guidebook to assist in consolidating self-assessment ratings of each immediate outcome determined in the Guidebook. It was envisaged that these would be transcribed into MS Excel to produce a consolidated numeric and graphical view of progress that may be useful for reporting, strategy, and planning in the future. Reporting periods can be compared and contrasted to track change over time
	Both tools can be used to generate new knowledge for the system or set of systems in which actors and agencies are embedded, making them systematic to use, and providing a systemic analysis by weaving the multiple systems and their interactivity for as comprehensive and consolidated a view as possible

Table 2.

Definition of the six key areas used in the M&E Framework.

1. Trafficking in persons awareness and prevention	Trafficking in persons prevention strategies, including public awareness campaigns that aim to prevent and combat trafficking in persons, especially against women and children, and to ensure just and effective punishment of traffickers
2. Victim identification	Victim identification is the process by which an individual is identified as a victim of trafficking in persons, which in turn, entitles them to their rights and assistance and protection services, such as medical and other health services, accommodation, food and basic needs, counselling, and psychosocial care among others. Accurate and timely identification facilitates the rescue and the rapid provision of care of a victim. It disrupts the trafficking process and supports the prosecution of the perpetrators
3. Reflection, recovery, and shelter	A recovery and reflection period allows victims time to recover from their experience. It helps them to start to re-establish their physical, psychological, and mental and social well-being. Shelter or temporary accommodation facilities provide a safe and secure environment for the trafficked person to gain access to essential services, to recover, and to assist with any legal proceedings. During this period, victims are able to properly process all the information and options available to them
4. Return and reintegration	Returning victims to their point of origin should be safe and dignified, with due regard for their privacy and anonymity. The return of any victim should be voluntary. Victims who wish to return home should be assisted in doing so safely without undue or unreasonable delay. For those who are not able to return home, especially if that could endanger their safety and well-being, alternatives should be explored
5. Medical and mental health care	Right to health is a basic human right. For a victim of trafficking, having access to medical and mental healthcare is critical. Most of them have health problems that are minor or severe. Their physical and mental health may be damaged through conditions of exploitation or direct assault as a control measure. Some may have contracted illnesses or become physically disabled during trafficking. Women and girls who have been sexually exploited may contract sexually transmitted diseases and/or become pregnant
6. Criminal justice processes	Victims’ rights and needs should be at the core of the whole criminal justice process. Yet victims are regularly objectified and regarded as a piece of evidence within a criminal justice process that centres on the prosecution of offenders. The ASEAN Regional Guidelines and Procedures affirms that all victims of trafficking participating in the investigation or prosecution of their exploiters should be provided with adequate, legally mandated protection, support, and assistance for the duration of their involvement in criminal proceedings. ASEAN countries are required to ensure that victims are aware of available remedies and to put in place the necessary laws and procedures to ensure the possibility of compensation being obtained

Consultation and end-user participation

The value of consultation and frequent end-user review of documents in development was continuous review of the boundaries around our key M&E components and core human trafficking concepts. The outcomes and indicators of the three guidance documents were disaggregated into an M&E Framework matrix in Microsoft Excel. From this analysis the components of the logic model emerged and the draft outcomes and indicators cyclically and simultaneously reviewed through desktop analysis and iterative consultation with stakeholders. Their perspective and critique identified emerging themes, found duplicate themes, contradictions, or anomalies.

The consultation process ensured our team that the emerging framework was meaningful, measurable, and useful to frontline responders working across a range of settings in ASEAN’s ten-member states. The extensive consultation revealed a diverse range of perspectives on the material, and the relationships and interactions between actors regarding how it would ultimately be used. Differing priorities were captured and resolved through their integration into the M&E Framework matrix.

We were mindful of the power dynamics between stakeholders providing multiple forums and ways of working to overcome barriers and ensure as many voices and perspectives were heard. Workshops, focus groups and key informant interviews were conducted with 270 participants of which 26% of individuals participated more than once in the consultation process. A total of nine consultative workshops were conducted with stakeholders, including senior representatives of the 10 member states. Identified and anonymous feedback on draft manuscripts were also provided in a quasi-semi-blind review process.

Diversity and representation of marginalised groups guided many questions of our participants for their perspectives on the suitability and accessibility of the work. The Gender, Environments and Marginalised Voices framework helped us define who those marginalised groups were, and how they were to be named. This went further than conducting a sex disaggregated audit of workshop attendees and ensured the issues and tensions concerning the naming of ethnic, religious or sexual identities was appropriate and acceptable across the ten-member states.

Next steps – change management

The monitoring of immediate outcomes under this project will be an important first step and we see the opportunity for the Commission to capitalise on frontline worker’s engagement and drive uptake of systemic change from the grass roots. The development of the Guidebook was a strategic decision used to support end-users to think systemically, facilitate dialogue within their organisation and with others and reflect meaningfully on their work. As mentioned above, implementation of the Guidebook and accompanying tools are being piloted and therefore results are beyond the scope of this article. We believe, however, that the M&E system has the potential to be an important instrument for the transformational change and institutional development of counter human trafficking practice for ASEAN member states.

That being said the Guidebook has limitations. The Guidebook lacks notes on how to analyse consolidated results of self-assessment. There are also gaps about monitoring of intermediate outcomes, their measures of success and reporting formats. Drawing inspiration from Gisela Wendling and David Sibbet (2018), we proposed a three-tier approach to foster a culture of monitoring, evaluation, and learning in counter trafficking in persons. At Tier 1, and currently in pilot, the Commission’s role is to translate, publish, and disseminate the Guidebook, with frontline workers primarily focussed on intra-agency monitoring and improvement using the provided tools. The benefit to the end-user being tested is intra-agency monitoring and improvement by using the self-assessment checklist. This may be a useful tool for a range of organisational activities such as annual reports or project grant acquittal, and may stimulate internal dialogue promoting organisational change to better respond to counter trafficking.

Tier 2 introduces a more systemic perspective, emphasizing dialogue, shared learning, collaboration, and cross-sectoral practice. ‘Change Champions’ might be engaged to lead efforts to maintain and expand engagement and coordination achieved during the Guidebook’s development, leveraging their influence to foster collaboration and knowledge-sharing.

Tier 3 involves central focal points or similar organizations driving systems change, coordinating data collection, and integrating self-assessment into existing reporting cycles. This tier requires substantial long-term investment to promote systems change, coordinate data collection efforts, and instil a commitment to M&E at the national and regional levels. The three-tier approach lifts the perspective of the M&E system from its component parts to a holistic view.

Lessons learned

This project has provided rich opportunities to further our knowledge and practice of systems thinking for evaluation. The authors have conducted program evaluations using the Inclusive Systemic Evaluation (ISE) Approach for Gender Equality, Environments, and Voices from the Margins approach but this was their first opportunity to apply the inclusive systemic evaluation and Gender, Environments, and Marginalised Voices frameworks to the design of an M&E system. The authors have experience developing M&E frameworks for complex and wicked problems, but they had not previously applied systems thinking principles or methods to those assignments. In the following section, we share our learnings to help the reader to plan, prepare, and apply a systemic thinking lens when designing an M&E system.

Understand the problem

Initially, the assignment appeared straightforward, focussing on leveraging three guideline documents to address counter trafficking in persons through M&E practices. However, this view oversimplified the task and a shift in our mindset was required. The complexity of counter trafficking, spanning cultural, geographic, and political boundaries of ten ASEAN member nations, along with external influences, required extensive consultation. Furthermore, human trafficking continually evolves and changes as perpetrators and organised crime recruit new victim profiles to match new opportunities created by changing technology. Recognizing trafficking as a wicked problem fundamentally altered our understanding of project scope and interactions, and how we believe stakeholders need to be viewing this problem too. This experience highlighted the necessity of acknowledging the complexity of wicked problems from the outset, informing subsequent adjustments in approach, seeking multiple perspectives throughout the project, and writing a practical, systematic guidebook and tools to foster systemic analysis, performed by people who may or may not have M&E skill sets, or an awareness of systemic evaluation.

Understand the component parts and how they interrelate

A characteristic of a wicked problem is that it cannot be solved with one intervention. The end-users of the M&E system confirmed that counter trafficking frontline responders need multiple pathways to engage with the M&E system whatever their level of competence, expertise, or field of practice. One way we did this was to establish the Building Blocks and Maturity Framework as foundational to the Guidebook. It centred the Guidebook as a guidance for monitoring present practice and allowed actors from every element of counter trafficking practice to learn, evidence, and share their skills and knowledge.

Upon reflection, we have realised that this was achieved after extensive analysis of the intersections of multiple systems. In better understanding the problem we were able to formulate in our minds a boundary around the counter trafficking practices, skills, and knowledge of our stakeholders, we gained insight into the governance structures at national, regional, and local levels, and the political tensions surrounding these. Understanding these formal and informal interrelationships was essential and informed the design of the Guidebook as a strength-based self-assessment tool to support effective reporting and to ensure the sustainability of the M&E system without additional resources. Thus, our task evolved into deciphering the complex dynamics between these systems to shape the design and implementation of the M&E framework.

Understand the audience

A characteristic of wicked problems is that of diverse and multiple perspectives held by stakeholders. For a problem like trafficking in persons there is an inexhaustible number of potential interrelated stakeholders with varying views and situated perspectives. We needed to set about understanding the needs and circumstances of the work’s intended audience and why practitioners may have been reluctant to engage in counter trafficking in persons monitoring and reporting in the past. These considerations included, lack of resources for monitoring coordination and support, wide range of user experience and skills, differences in operational focus for various agencies, internal and external funding dependencies, and political factors.

Another key learning was our under appreciation of the full measure and intense interest in the project, generated by people in government and civil society across the ASEAN member states. Clarifications with the project managers regarding the audience for the work and the consultation strategy required expanded the boundaries to include more participants, and changed our practice to conduct more face-to-face workshops and focus group meetings. As reported above, we engaged with a wider number of informants and diversified our strategies to engage with them meaningfully which improved the quality of the project’s deliverables and its overall success. In fact, such was the determination of people to participate, a participant in Bangkok arrived in an ambulance to circumvent gridlocked traffic conditions and arrive on time! Endorsement by all ASEAN member states of the final products was swift and non-contentious with acknowledgement of their quality and reliability due to the wide representation of the diverse sector.

Understand our place in it

Our readers may understand the feeling of sinking into and becoming immersed in a system in which one may initially have been objectively detached. Recognising the moment in which this occurs is often difficult. As systemic thinkers we can, and should, however, expect this to happen as once we begin to work in a system, we become embedded in the system (Midgley, 2023). No longer outsiders looking in, we had to accept an unwitting participation in politics and agents’ use of the project to leverage other interests, beyond our control, influence or anticipation. Fortunately, our relationships with key people in the systems, particularly our clients or commissioning agents of the project were supportive, available and open to regular, honest dialogue about the work, our place and those responsible for managing various emerging issues. New parts of the system, relationships with others and the sub-systems in which others are embedded become ours, not just to know but to navigate almost daily. While most good evaluation practitioners would concur with this responsibility, the difference here is that in complex and wicked problems, people come in from the side lines, unexpectedly carrying influences that change the dynamics suddenly, even if this comes from other projects or agendas that we find now intersect with ours. This may include another project commissioned by the same commissioning body, a structural change to a key agency such as a rotating chair, or a report on counter trafficking in persons progress that affects every ASEAN nation produced by an external national government. All events carry knock-on effects that require immediate comprehension, reflection, a redrawing of the boundaries, consideration of our role and influence (and that of others), and flexibility to respond.

Understand there is a gulf between conventional and systemic thinking paradigms

As systemic thinkers we know that we need to be flexible and adaptive and to anticipate emergence. We also prepared training, induction, and capacity development of our own team (3 in-country consultants), all of whom are familiar with evaluation but have more varied experiences with systemic thinking. This alone may not be enough as an investment of time to support people to work with systemic thinking concepts and ideas, may also be necessary. Two learnings here is that push back against systemic processes can present major challenges and, in our case, required instantaneous response that may be daunting for the novices in this field, and the extent to which the conventional logic model of highly sequenced and predictable in-put/out-puts is entrenched in practitioners’ minds, even in settings where agents or agencies do not have a strong culture of applied M&E.

As part of our learning above regarding understanding the needs of the end-users of this work, we now realise the extent to which people arrived at our workshops without having, or having had very little exposure to any alternative to the classic logic model. While the logic model presented to the participants was familiar in its conventional format, the questions and sequence of the workshop were grounded in the systemic approach that we brought to the design of the M&E System. In some instances, this was very unfamiliar. One workshop was very nearly derailed over this issue. Our in-country consultant renegotiated to enable the workshop to continue, with modifications to the sequence in which the discussion unfolded. We might have avoided this by learning ahead of time stakeholder’s understanding of evaluation. On the flip-side, the facilitation of our second round of in-country consultations was led by the ASEAN-Australia counter trafficking team as expertise in M&E did not translate into expertise on human trafficking.

When working to uncover diversity and inclusion of marginalised people and communities, some discussions were shut down quickly by people in leadership, providing us a top-down list of ‘these are those who matter’ rather than allowing the voices of people closer to the ground to talk to us about who they might like to include. This also reflects a difference in the mindset between ourselves and some of our consultees. On reflection, perhaps clearer communication could have helped them to understand that we were not wanting to apply a checklist approach to gender and diversity inclusion. We were digging to find the nuances between social circumstances that make people more vulnerable to trafficking in ways that frontline responders might see from their experience, but not easily categorise on a list. Reflecting on these events, however, reinforced our commitment to designing simple, systematic, procedural tools and processes that included ways to think more deeply about gender and diversity, resulting in an accumulative shift towards systemic and holistic monitoring and assessment practices. In future, we will also spend more time exploring systemic thinking concepts and how to work with systemic thinking concepts and ideas, with our in-country consultants.

Understand that the system will evolve

There is a real opportunity here to nudge transformational change through the practice of monitoring, sharing, co-learning, and evaluation reporting on counter trafficking in persons. Transformational change is achieved by encouraging new systems, rather than recommending incremental change to the current system. The new system can then better address the multiple dimensions of the wicked problem. Transformation is another way to view impact. As Daniel Kehrer (2020) states: ‘transformative change converts a current ecological, social, political, economic, scientific, or technological system or all systems together into a fundamentally new one that, from there on, forms the new mainstream’ (para. 9). This has to be an end-goal when working with any wicked problem.

Conclusion

As we draw this discussion to a close, we conclude with some reflections on our approach to this assignment. Our application of the Inclusive Systemic Evaluation (ISE) Approach for Gender Equality, Environments, and Voices from the Margins systemic evaluation concepts, such as the Gender, Environments and Marginalised Voices framework, developmental, action learning, and participatory design were highly effective to collect quality data, synthesise multiple inputs, sense-make the various components of the system, and engage in authentic participatory consultations that were adapted for inclusion, availability, accessibility and language. The need for a nexus between the application of systematic approaches and structures to facilitate a systemic perspective by end-users was made possible by this approach. And while the Inclusive Systemic Evaluation (ISE) Approach for Gender Equality, Environments, and Voices from the Margins guidance was not written as a guidance for work of this type, its key concepts including systems thinking, boundary critique, reflexivity, and the Gender, Environments, and Marginalised Voices framework, emergence, and concern for power and ethics, were guiding principles for our project management.

The project has reinforced the value of systemic thinking for evaluation, particularly as it pertains to wicked problems. Trafficking in persons may be a wicked problem less well tackled due to conflicting social values, norms, behaviours of governments, and high level-systems, with differing views on policy, operational procedures, funding commitments, and timeframes for cooperation with conventions such as the Convention. Our task, in a nutshell, meant that the M&E Framework and associated tools would need to be relatively simple and inexpensive to administer, flexible enough for users to adopt whose role may relate to some but not all key areas, and be strength, rather than deficit based. We delivered a coherent streamlined system that is aligned with ASEAN priorities and ASEAN-Australia counter trafficking’s desired outcomes that could be used by conventional M&E practitioners and those without any background in systems thinking. The Guidebook emphasizes equity and inclusion by focussing on marginalized groups, maintains a consistent approach to monitoring and reporting, and ensures that entry to the M&E system is accessible from multiple pathways at varying levels of organisational or institutional progress to monitor a very wicked problem. This simple guidance holds a powerful promise: with ongoing systemic support, it situates the role of M&E practitioners as leaders in the long-term agenda to eradicate human trafficking. While this may be an emergent project outcome from some perspectives, we think the people commissioning this project have more far-reaching ideas and we look forward to the outcomes of the pilot project.