Benefits of reflection-based monitoring in action research projects

Abstract

This article reports on reflection-based monitoring tools developed and used in an action research project to improve gender equality in science and research institutions. Several of these tools were developed to facilitate individual and joint reflection from the perspective of the co-researchers involved. The aim of the article is to make the case for the integration of a systematic reflection-based monitoring approach, to describe its approach and theoretical underpinnings that link critical reflection theory with reflexive action learning theory, and to present the details of six monitoring tools so that they can be adopted in other project settings. It also highlights how these monitoring tools stimulated both individual and group reflection and how they facilitated the sharing of knowledge and experience between change agents involved in co-production. The article also shows that the process monitoring approach based on systematic reflection can contribute to successful project implementation and offers several important advantages.

Keywords

Monitoring reflection action research gender equality organisational change

Introduction

Monitoring is often understood as a tool for tracking the implementation progress of a project or programme in an organisation based on pre-defined, most often quantitative indicators. Monitoring tools can, however, also be used to facilitate reflection among change agents. In this article, we introduce the systematic reflection-based process monitoring approach that we developed while working during four years in an action research-based project, funded by the European Commission. We explain the theoretical foundations of this approach and describe the reflection-based monitoring tools which we designed to support individual and collective reflection among the change agents involved (cf. Dahmen-Adkins & Peterson, 2021). The research project that our work builds upon focused specifically on increasing gender equality within seven research performing organisations (RPOs) through the design and implementation of tailor-made gender equality plans (GEPs). We however argue that the reflection-based monitoring approach offers several important advantages for action research projects in general: it creates space for mutual exchange and observation; it increases both individual and organisational capacity building and learning; it contributes to mutual encouragement and empowerment of change agents; and it can improve the interventions and actions and the implementation of the change plans.

We thus hope that the reflection-based monitoring tools developed in our action research project can be adapted and adopted in other project settings.

The article continues by situating the topic of monitoring in action research. This is followed by the theoretical framework combining critical reflection theory with action reflection learning theory. The subsequent section explains the systematic reflection-based monitoring approach, situated in the project context in which it was developed. The article’s main section describes six reflection-based monitoring tools in detail, emphasising the dimensions that facilitate and promote reflection. A brief discussion and summary conclude the article.

Monitoring in action research projects

Action research is an integrative approach that combines theory with practice, focusing on addressing real-world issues through a synergy of researchers and practitioners. This dynamic approach operates within a set ethical framework (Avison et al., 1999) and hinges on iterative cycles, each condensing a series of interconnected activities, starting with the identification and diagnosis of a problem, followed by the formulation and execution of an action plan (or intervention), and culminating in a period of reflective learning (Coghlan, 2019). The process does not end here; post-reflection, the cycle embarks on another round, where insights from the prior phase guide the subsequent steps. Thus, action research continually involves data collection, feedback, analysis, action planning, implementation, and evaluation, each ideally leading seamlessly into the next, fostering an environment of perpetual learning and problem-solving (Coughlan & Coghlan, 2002).

To execute action research effectively, data must be analysed and confirmed in alignment with the deliberate progression of the action research cycle (Maestrini et al., 2016). Thus, a systematic and methodical monitoring plan can be regarded as essential for the success of an action research project (Neumann et al., 2018). Monitoring enables the assessment of the project’s progress, identification of processual obstacles, and facilitation of necessary modifications to meet the project’s goals (Kalpazidou Schmidt & Cacace, 2018; Kusek & Rist, 2004). By tracking and reporting progress at different stages of the project, monitoring ensures that the desired outcomes are achieved (Chen & Rossi, 1989; Funnell & Rogers, 2011; UNDP, 2009). Monitoring, in this sense, overlaps with certain definitions of formative, process, or developmental evaluation (Haylock & Miller, 2016; Neumann et al., 2018).

An effective monitoring strategy typically involves the collection of a variety of quantitative and qualitative data, using different methods such as surveys, focus groups, and participant observations (Scriven, 2013). Which monitoring tools to adopt depends on data’s relevance and reliability as performance measures (Pawson & Tilley, 2001). In action research projects, such data comprise observations of participants’ actions, their learning trajectories, and the consequent influence of that learning on subsequent actions (McNiff, 2016; McNiff & Whitehead, 2005). The participatory aspect of monitoring emphasises the dual function of individuals as both essential members of organisations and drivers of change (Coghlan, 2019).

Participatory monitoring tools further enhance engagement and commitment among participants (Friedman & Rothman, 2015; Gatenby & Humphries, 2000). The knowledge obtained through monitoring assists in improving interventions, facilitating plans for change, and increasing the motivation of participants, ultimately supporting both organisational and individual learning (Kusek & Rist, 2004; Patton, 2011; Reid et al., 2006; Riedy et al., 2023; UNDP, 2009). Knowledge is generated when researchers and participants reflect on change processes and gain a greater and improved understanding, which may lead to revisions of initial action plans (Reason & Bradbury, 2008). This suggests that monitoring tools should collect data through people’s feedback on how learning and understanding are developing and through self-reflection exercises with change agents (McNiff, 2016).

Critical reflection theory and action reflection learning theory

The article draws on a theoretical framework that combines critical reflection theory with action reflection learning theory, which both take their starting point in reflection as a human mental activity and cognitive process involving introspection, meaning-making, problem-definition, and forward-thinking (Høyrup, 2004; Rimanoczy & Brown, 2008).

Reflection entails considering experiences, feelings, or incidents and systematically pondering their meanings, interpretations, and implications for future actions. This cognitive process often suspends conclusions through in-depth inquiry and involves ’turning back’ on various experiences and events based on awareness of perceptions, thoughts, actions, and habitual responses (Dewey, 1910[1997]; Mezirow, 1998). Daudelin (1996) proposes a distinction of four stages in this reflection process: (a) articulation of a problem; (b) analysis of that problem; (c) formulation and testing of a tentative theory to explain the problem; and (d) action (or deciding whether to act). Boud et al. (1985) similarly identify three core elements in the reflective process: returning to experience, attending to feelings, and re-evaluating experience.

When the object of the reflection is one’s abstract thinking, assumptions, presuppositions, and the grounds of one’s belief, we are concerned with what can be labelled critical self-reflection (Mezirow, 1998). Critical self-reflection is a systematic, rigorous way of thinking that involves formulating and understanding critiques of the premises of assumptions and thinking. Critical self-reflection also involves applying wisdom gained through experiences, which offers the potential to produce a change in one’s established frame of reference and result in new competencies and new solutions to problems and future actions (Høyrup, 2004; O’Neil & Marsick, 1994). Such critical self-reflection may arise from a rapidly changing and turbulent environment or external challenges in an unpredictable world with complex problems where we have to deal with situations of uncertainty and instability (Daudelin, 1996).

Although critical reflection is a personal cognitive procedure, it is always situated temporally and spatially, reinforced by others, and embedded in social interactions (Dewey, 1910[1997]; Høyrup, 2004). Thus, reflection can be facilitated by a group environment where members can share their different perspectives and experiences (Rimanoczy & Brown, 2008). This sharing can help others reflect on their challenges and find new solutions and ways to deal with them (Daudelin, 1996).

Reflection and learning are thus closely related, and reflection is essential for the continuity of learning (Dewey, 1910[1997]) and personal and professional development (Schön, 1983). It is also a crucial feature of a type of organisational learning referred to as triple-loop learning, which involves complex problem-solving and continual reflections on the organisational context and functioning, resulting in transformational change of organisational practices, norms, and values (Tosey et al., 2012). This type of organisational learning is a key feature in many action research projects, particularly so in international development programmes with a feminist or gender perspective. These action research projects are often characterised by a participatory approach which focuses on capacity-building and feedback mechanisms, which can inform ongoing program adjustments and contribute to learning and improvement (see e.g. Bamberger & Podems, 2002; Espinosa, 2013; MacArthur et al., 2022; Podems, 2010).

When reflection is used systematically as a methodology in an action research project, it benefits from being guided by different tools for feedback, planning, and assessment (Rimanoczy & Brown, 2008). The context in which monitoring tools for such guided reflection (and learning and action) were developed is described below.

The context of the gender equality action research project

The project setting

This article is based on experiences from a cross-cultural implementation project funded by the European Commission for three years. Seven RPOs from seven European countries participated in the project, which aimed to initiate and sustainably institutionalise structural change towards gender equality through tailor-made GEPs. The project followed the four phases of the action research cycle: 1. Plan, 2. Act, 3. Observe, and 4. Reflect (Coghlan, 2019).

Planning (phase 1) began during project proposal preparation by analysing the current state of gender equality in each institution and proposing interventions and areas of action for the GEPs. The initial six months of the project then centred on the creation of the final version of the GEPs to be implemented. Interviews and discussions conducted during this planning and preparation phase with actors from each institution provided an opportunity to engage stakeholder groups in the process of change at an early stage of the project (cf. Espinosa et al., 2016; Vaughn & Jacquez, 2020).

In the second phase, the change agents in the implementing RPOs started to take action by introducing the intervention measures in the GEP in their organisations.

The monitoring tools for guided reflection could develop their full potential during the observation phase when they were utilised to critically assess the implementation process of the respective GEPs. These tools were essential not only in structuring and focusing the observations but also in interpreting and comprehending them. Phases 3 and 4, the reflection phase, thus had a certain overlapping potential, as the observations led to critical reflections, which led to a further iterative development process of certain gender equality measures that could not be successfully implemented for identified reasons.

The project members

The importance of working collaboratively to achieve long-term and sustainable transformative change was emphasised in the project, in line with a feminist participatory action research (FPAR) approach (Frisby et al., 2009; Reid et al., 2006). FPAR favours: “Methods which emphasize collaboration and dialogue as appropriate to the community” (Gatenby & Humphries, 2000, p. 90) while promoting self-reflection and empowerment. This implies shifting the focus from research on people in organisations to research with people in their organisations (Coleman & Ripping, 2000). For a FPAR project, it is essential to establish trust and actively engage as co-researchers both gatekeepers inside the organisations and target groups for the actions in the change project (Coleman & Ripping, 2000; Pope, 2020). This was a significant undertaking for the national core teams in the action research project detailed in this article, which consisted of two or three key change agents, usually of different ages and at different stages of their careers. All had worked at the respective RPOs before the project began and were familiar with their structures, which entailed various overlaps in their roles: they were both organisational insiders and action researchers. Additionally, some national core team members were also the target audience for their gender policy changes implemented through interventions.

The core teams were supported by so-called transfer agents (TAs), individuals distinguished by their institutional roles, who internally facilitated the GEP implementation and dissemination of project results (Thaler, 2016). Often from the management or gender equality sectors, these TAs backed the project, offering access to gatekeepers through unique intra-institutional networks. Their involvement was strategized to counteract resistance to institutional changes, foster joint ownership (Donnelly & Morton, 2019), and ensure the sustained importance of gender equality issues (cf. Coleman & Ripping, 2000).

The monitoring team was constituted by us, two project members, with one fulfilling a dual role as a member of both the monitoring team and the national core team. This position provided us with first-hand and indispensable insights into the implementation process, which heightened our awareness of the implementation challenges and ensured that the monitoring tools were developed while taking these challenges into account.

The monitoring approach

During the first six months of the project, the focus was on developing techniques to monitor implementation procedures in institutions. It is worth noting that the project differentiated between monitoring the implementation process of the GEPs and assessing the results of the measures and activities implemented in the GEPs (cf. European Commission, 2012; OECD, 2002; Holbrook & Frodeman, 2011; Lipinsky & Schäfer, 2016). This article centres on the first aspect, exploring the benefits of process monitoring. Evaluation in the second sense was addressed separately, though both tasks overlapped and were used in a synergistic way when possible.

Developing the monitoring approach necessitated strategic and informed decisions regarding what, who, and when to monitor - based on theories of change and the mechanisms of intervention programmes (cf. Rossi et al., 2004). The timing of monitoring was linked to the different stages planned for the interventions to be implemented. Another crucial prerequisite for implementing an extensive monitoring plan was the allocation of sufficient funds to cover the responsible parties and project partners. This enabled process-oriented monitoring to take place from the beginning of the project until its completion. Appropriate resources have also been provided to develop the monitoring capacity of implementing organisations to support their sustainability after project completion.

Some of the most important decisions to make regarding the monitoring activities involved the data to be collected. The project utilised a mixed-method approach, with a preference for qualitative data on the implementation of the GEPs as it was acknowledged that producing such information would necessitate ongoing reflection among project members.

The generation and collection of monitoring data took place primarily during the regular face-to-face project meetings, which were used for consortium-wide exchange and reflection. Coordinating monitoring at these meetings both saved resources and placed monitoring at the heart of project activities. Typically, these meetings were organised every six months and the meeting agendas were carefully designed to allow sufficient time for monitoring. Individual or team-based self-assessments were conducted by participants within their usual work environment and between meetings to supplement group monitoring sessions.

Determinations on monitoring specifics and the creation of new tools were based on project observations, including participants’ needs and assessments of previous monitoring results. For this purpose, participants were specifically asked for feedback after each application of a tool: What was considered helpful, what needed to be deepened, and what could be improved? This approach thus places less emphasis on pre-established indicators and ensures adaptability to partners’ needs and changes.

Systematic reflection-based monitoring

Generally, monitoring is defined as the systematic collection of information regarding programme implementation with the use of monitoring tools deduced from programme objectives. However, with systematic reflection-based monitoring, the focus of the monitoring tools shifts from primarily concrete actions and practises to reasoning, reflection, deliberation, and contemplation - about actions and practises but also about one’s situation and position (cf. Høyrup, 2004). In the presented case, it involved analysing how formal and informal factors impacted GEP implementation in the RPOs. This process was facilitated by the six reflection-based monitoring tools introduced in this article, which together covered all stages of the reflection cycle described by critical reflection theorists (Boud et al., 1985; Daudelin, 1996; Høyrup, 2004): articulation of a problem, analysis of that problem, formulation and testing of a tentative theory to explain the problem, action (or deciding whether to act), feedback processes, processes of thinking ahead, returning to experience, attending to feelings, and re-evaluating experience. The application of the different tools in different co-productive settings enabled the national core teams to access the different stages of the reflection process individually and collectively.

Graphic 1 below provides a comprehensive overview of the six reflection-based monitoring tools described in this article. These tools have been specially designed to support and facilitate the various iterative steps within the action research cycle of a gender equity change project. Each tool is tailored to support reflective practices at different stages - from individual to team reflection, and from understanding to evaluating change.

Graphic 1.

Overview of reflection-based monitoring tools: Their purposes and role in stimulating iterative steps in the action research cycle.

This graphic not only outlines the specific purposes of the tools used within the action research cycle but also provides a clear framework for catalysing iterative reflection. It can serve as a visual facilitator to guide practitioners through a structured process of monitoring and evaluating progress, challenges and successes in gender equity initiatives. Together, the tools ensure that each phase of action is linked to strategic analysis and refinement, fostering an environment of continuous improvement and adaptive learning.

Introduction of the applied reflection-based monitoring tools

The six reflection-based monitoring tools are further described below. The description of each tool focuses on the specific dimensions designed to encourage individual and collective reflection.

Exchange workshop

The qualitative monitoring tool Exchange Workshop was implemented at the end of the first project year after all implementing partners had gathered their first experiences related to the practical implementation of the GEP. A framework based on organisational change and feminist participatory action research theories regarding the difficulties and oppositions to gender equality interventions (Frisby et al., 2009) was utilised in the development of this monitoring tool.

Purpose of the tool

○ To better understand change processes, problems and effective practices.

○ To collectively reflect on and exchange experienced successes and challenges.

Mode of data collection

○ Team reflection before the workshop: National teams had to submit a list of three challenges and three successes related to the GEP implementation to the workshop organisers.

○ The data collected was clustered and served as an entry point for the first part of the workshop during a project meeting.

○ Two workshop parts:

1. World café format to share experiences;

2. Work units for two paired teams each.

Outcome of the tool

● Each national core team developed a practical 6-month action plan based on prioritised challenges to be addressed with the support of another national team paired as a collaborative partner.

The Exchange Workshop aided in the identification and examination of problems encountered during a change process. A significant realisation from the tool was that change agents often concentrate on problems and resistance during implementation, disregarding achieved accomplishments. However, recognising and making visible such short-term wins (Kotter, 1995) is important to keep everyone involved motivated and to show organisational members and stakeholders that change is taking place. The tool facilitated shared reflection on the experience, encouraging a shift in focus from obstacles to facilitators and successes. This, in turn, created a feedback loop that replaced negative feelings with recognition of successes. The result was a new way to think about the future and to look at future activities more constructively. On a monitoring level, the results also contributed to the development of a specific tool focusing on incremental change (see below).

Operational process monitoring tool

Similar to the Exchange Workshop, the Operational Process Monitoring Tool was also implemented at the end of the first year of the project. The tool’s purpose was to stimulate the implementing national core teams to reflect on specific challenges related to the attempts to achieve specific objectives of the GEP. The development of this tool was based on previous action research on feminist interventionist change initiatives, with a particular focus on organisational power and management frameworks. Research literature shows that gender equality interventions can face varying degrees of resistance depending on their thematic focus (van den Brink & Benschop, 2012).

Purpose of the tool

○ To monitor the process of implementation towards the GEP’s objectives.

○ To specifically identify challenges and success factors related to the objectives.

○ To reflect on reasons for cancelled actions or major revisions of actions.

○ To provide a common space for the national teams to discuss and reflect.

Mode of data collection

○ Reporting template to be filled out per national team.

○ The requested information was structured around the objectives stated within the GEPs.

○ The tool was similarly tailor-made for each national team and allowed the teams to include information about the status of specific actions.

Outcome of the tool

● The teams identified both key success factors, like effective collaborations and management support, and problematic hindrances, such as informal practices and limited gender awareness. The tool allowed for the teams to articulate and reflect on the use and usefulness of various strategies to overcome challenges related to time, staff, and funding. It also highlighted that the teams sometimes had to modify or replace actions in the GEP. The tool’s results were vital for understanding progress at early implementation stages.

The strength of the Operational Process Monitoring tool was that it promoted a participatory approach to developing and testing preliminary theories to explain the problem(s), based on individual practical experiences and a collective re-evaluation, intending to improve and advance future work. This tool enabled change agents within national core teams to devise multiple strategies for tackling and responding to implementation challenges within institutions. As a result, a process of co-creation was set in motion, fostering mutual learning and contributing to capacity building among the team members.

Self-assessment of change agent role

The Self-assessment of Change Agent Role monitoring tool was introduced in month 24 of the action project, exactly at the “halfway point” of the project, a critical phase in which obstacles need to be identified and corrections made. This tool was inspired by research-based theories about organisational change and gender relations (cf. Pawson & Tilley, 1997) and by interviews with experienced evaluators, experts, practitioners, and change agents (cf. Brouselle & Champagne, 2011; Sridharan & Nakaima, 2011), who emphasised the importance of also monitoring the people who actively implement GEPs.

Purpose of the tool

○ To document individual factors that motivate change agents to engage in organisational gender equity work.

○ To provide a personal perspective on success factors and challenges, especially resistance.

○ To collect individual memories of positive and/or negative reactions to their attempts to improve gender equality in their organisations.

Mode of data collection

○ A template was distributed consisting of six questions, considered to be the most relevant for collecting data about the experiences of change agents for gender equality.

○ Individual self-reflection by every member of each national core team.

○ Each team member was encouraged to write down as many reflections as possible.

Outcome of the tool

● The guiding questions in the template prompted national core team members to self-reflect on their experiences and consider structural factors and resource needs. It aimed to empower change agents and foster solidarity among the core team members. The monitoring teams compiled and shared these insights with the project team to increase awareness of appropriate resources and support factors. A concrete result of the tool was the identification of six practices for successful change agency, involving or presupposing systematic reflection: 1. Communicating; 2. Community building; 3. Building trust and legitimacy; 4. Accumulating and using resources; 5. Using and transferring knowledge; and 6. Drawing on personal motivation.

This tool produced data that supplemented the information gathered with the help of the previous-mentioned tools by focusing on individual narratives from the change agents through a re-evaluation of their experiences, which also presupposed processes of thinking ahead, returning to experience, and attending to their feelings as change agents and the feelings among members of their organisations. The tool thus involved critical self-reflection by each member of the national core teams, but it also widened the perspective to the surrounding institutional context and supportive and hindering factors within this context.

Peer consultation reflection session

The Peer Consultation Reflection Session monitoring tool was developed building on the results of the two previously described tools, reflecting an important approach to the underlying concept of continuous, sequential monitoring: developing tailor-made tools by observing and reflecting on the results of the methods used so far (cf. Haylock & Miller, 2016). The Peer Consultation Reflection Session was implemented in month 30 out of 48 of the project. The development of the tool was a result of ongoing observations of project activities (cf. UNDP, 2009), and a space for knowledge sharing was something that project members had requested and previously valued. In addition, the tool was inspired by previous literature on reflection exercises in peer consultations (Brown et al., 1999).

Purpose of the tool

○ To create an arena for shared reflection on the GEP implementation process.

○ To discuss experienced challenges and resistance and to stimulate joint problem-solving among peers.

○ To encourage the change agents to see their situation from a new point of view and to understand problems differently.

Mode of data collection

○ Workshop concept for the national core teams.

○ Participants were divided according to eight themes and whether they were able to give constructive advice based on their previous experience with the topic or whether they would need or could use such advice in the near future.

○ Participants switched between the role of presenter and reflection facilitator.

○ Final discussion on insights within the whole group.

Outcome of the tool

● The tool yielded insights across eight themes, highlighting challenges such as difficulties in creating indicators to reveal inequalities and resource shortages for data collection. It suggested utilising existing databases in collaboration with HR departments as a solution. Discussions also focused on women’s networks, addressing issues in engaging women in networking and enhancing the networks’ impact. Shared experiences included resistance from men to these networks and strategies for fostering empowering mentoring situations.

In particular, this tool enabled the workshop participants (in this case, the national core teams, but divided into mixed, thematic groups) to reflect on, articulate, and thoroughly define the problems they had encountered in implementing their GEPs, while at the same time using the support of their peers to reflect together on workable solutions. The participants were divided into thematic groups based on pre-identified similar challenges, which were used to create discussion topics. These challenges and discussion topics (a total of eight) were identified through the application of a previously implemented monitoring tool where the national core teams had provided feedback on the implementation progress (the so-called Interim Feedback Report, not included among the six monitoring tools in this article as it was not used specifically to facilitate reflection). This allowed for very focused discussion and reflection in the so-called peer reflection teams.

Incremental transformation monitoring tool

The Incremental Transformation Monitoring Tool sought to define impact relative to the starting point of each participating institution. It was the first tool to be implemented during the third and final phase, the reflection phase, and was designed to allow for reflection on the two previous phases. The design of the tool was based on evaluation theory, feminist theory, and policy research, emphasising the importance of including process criteria in evaluations of gender equality policies (Frisby et al., 2009).

Purpose of the tool

○ To facilitate a discussion about setting and achieving intermediate goals in a change project.

○ To reflect on context-specific achievements and their conditions for success.

○ To develop strategies to sustain the gender equality efforts after the finalisation of the project.

Mode of data collection

○ Workshop concept for all national core team members.

○ Division of the participants into three discussion groups related to different topics.

○ Each group documented their discussion in specific, tailor-made templates.

○ The outcomes of the small group discussions were fed back and discussed within the whole project team.

Outcome of the tool

● The workshop led to valuable insights and advice. It highlighted the evolution of the teams’ visions from basic to detailed and long-term, stressing the need for realistic and flexible goals and effective communication to drive change. Recommendations focused on avoiding high-resistance areas for initial interventions, leveraging small successes to motivate change agents, and emphasising personal benefits. Additionally, forming robust teams, integrating new agents for future planning, and prioritising networking were key strategies discussed.

The special feature of this tool was that, for the first time, participants were asked to actively reflect on the impact of the project within their organisations, to evaluate it jointly, and to think about the sustainability of the changes. The impact was not measured against a pre-defined indicator. Instead, a contextual and holistic understanding of incremental impact was developed through individual and collective reflection. A particular outcome of this tool was the change in vision associated with the intervention project at the organisational level. Throughout the project, based on these guided reflection sessions, the change agents’ vision of what could be achieved became more realistic, but also more detailed and long-term. This shows that the engagement and the resulting experience as a change agent at an individual level have triggered a processual learning process about change management within one’s organisation.

Most significant change technique

The Most Significant Change Technique was the last qualitative monitoring tool applied in the final month of the project’s duration. It was also one of the most comprehensive and complete monitoring tools developed since it involved several different groups of participants and also because the purpose was to facilitate long-term and broad reflection, including reflections on sustainable and transformative change after the end of the project. The tool is based on the most significant change methodology developed by Rick Davies and colleagues and is a qualitative tool based on stories (Dart & Davies, 2003).

Purpose of the tool

○ To collect participatory stories about change to monitor and evaluate the project’s impact.

○ To collect evidence of intangible changes, like behavioural and attitudinal changes.

○ To gather information about changes in both individual character and organisational character (but from an individual perspective).

Mode of data collection

○ The tool addressed the national team members as well as persons from the target groups of interventions at each organisation.

○ Dual mode of data collection using only two complex questions:

1) The written answers from target groups were collected and analysed; these results were fed back into a…

2) …workshop setting for core team members during the final project meeting. The workshop was divided into two parts, starting with individual reflection, followed by group reflection.

Outcome of the tool

● From the collected stories of target groups, a theoretical framework emerged, illustrating a typology of change dimensions: Individual/collective, informal/formal, and intangible/tangible. This framework and its dimensions were used to develop a workshop concept for the monitoring tool. Core team members were then asked to give examples of modifications within these categories. Individual and formal changes included greater involvement in decision-making due to recognised gender knowledge. Tangible and collective changes encompassed achievements like a gender-equal salary system, gender budgeting, improved gender balance in boards and committees, gender networks, and enhanced communications.

The tool enabled the collection of information about the outcomes and achievements of the change project that would otherwise have remained hidden if only quantitative monitoring tools had been used. It contributed to both individual and collective reflection and an improved contextualised understanding of the underlying project processes and impact. By providing a more comprehensive, contextual, and specific understanding of the barriers and facilitators to change, this monitoring data can enhance the sustainability of the project (Dahmen-Adkins & Peterson, 2019). At the same time, it supported an improved comprehension and reevaluation of cultural transformation, alongside the function of change agents in the change processes in their respective institutions.

Discussion and conclusions

We have argued for the importance of reflection-based monitoring tools to support triple-loop learning in transformative action research projects. The reflection-based monitoring tools presented in this article facilitated all stages of the reflection process (Boud et al., 1985; Daudelin, 1996; Høyrup, 2004). The tools were also designed to facilitate both individual and group reflections. Group reflections promoted the formation of diverse contexts for reflection. They let the change agents reflect within their core teams, with their target groups and stakeholders within their organisation, and with other change agents in the action research project. This allowed for different types of reflection, taking into account that the experience of action projects is very different for those involved in different roles.

The tools generated information important for sustainable and transformative change that would otherwise have remained hidden; for example, concerning various impromptu strategies and creative and iterative methods used by the change agents to manage complex and challenging situations and to make minor, but still key, adjustments to improve the implementation process.

The tools used in this project also provided crucial moments when expressions of solidarity and support were shared between the national core teams and change agents. Not infrequently, feelings of frustration, resignation, and disappointment can arise due to the recollection of experiences and the attention to feelings in both individual and collective reflection sessions. Revealing the collective nature of such individual experiences and feelings can lead to empowerment and renewed will and strength to pursue the transformative change agenda.

A systematic reflection-based monitoring approach can thus create co-productive settings within and between partners and thereby add valuable dimensions to the cycle of planning, action, and reflection in an action research project.

Conclusions and learnings

Drawing on the experiences from this action research project, we can conclude by summarising our most important learnings concerning the implementation of a reflection-based monitoring approach and the development of reflection-based monitoring tools:

Include both individual and joint/collective opportunities for reflection.

Facilitate reflections within and between different actors involved in the project.

Consider the dual roles that change agents play in action research projects as both insiders and outsiders to the organisations they seek to change. It is essential to reflect on the perspectives of both insider and outsider characters, as they hold value in promoting the success of the project.

Introduce reflection sessions also in the initial phases of the action research project.

Use the reflection-based monitoring tools to support a systematic planning, action, and reflection cycle (cf. Coghlan, 2019) where regular reflection sessions are established as likewise important as planning and action.

Use formative evaluation to identify which reflection-based monitoring tools are most useful and relevant for the specific project. Be attentive to feedback from participants in reflection sessions.

Use an iterative approach to develop the reflection-based monitoring tools to tailor-make them for the purpose and assessment of the implementation processes.

Limitations and future research

According to Coghlan (2023), action research is “the social science of the possible” (p. 15). In this article, we have aimed to demonstrate how an organisational action research approach, focused on achieving the possible, benefits from a reflection-based monitoring strategy. The aim of the article has not been to go into details regarding the implementation of the GEPs and the long-term achievements at each of the seven RPOs participating in the project. Future research could to a greater extent take into account the particular focus of different action research projects and the possibilities of adapting monitoring tools to fit these different purposes. That would contribute further nuances to how reflection can enhance both short-term and long-term achievements in action research projects. The contribution of this article has been to introduce a systematic approach to reflection-based monitoring and report on how it was developed to provide spaces for reflection and facilitate a tactical discussion about what can be realistically achieved within the individual institutions and the given framework conditions.

Footnotes

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iDs

Jennifer Dahmen-Adkins

Helen Peterson

Author biographies

Jennifer Dahmen-Adkins is senior researcher at the Chair of Technology and Organisation at the Institute of Sociology at RWTH Aachen University. Her research focuses on gender and intersectionality research, as well as higher education and organisational research, each with a strong “theory-practice” connection.

Helen Peterson is currently a Professor in Sociology at Örebro University in Sweden. She has researched the Swedish higher education sector and the European science and innovation sector through a gendered lens for over 15 years. Her focus has been on academic management and leadership, women’s careers in male-dominated fields, gender mainstreaming and policy implementation.

References

Avison

Lau

Myers

M. D.

Nielsen

P. A.

(1999). Action research. Communications of the ACM, 42(1), 94–97. https://doi.org/10.1145/291469.291479

Bamberger

Podems

D. R.

(2002). Feminist evaluation in the international development context. New Directions for Evaluation, 2002(96), 83–96. https://doi.org/10.1002/ev.68

Boud

Keogh

Walker

(1985). Reflection: Turning experience into learning. Kogan Page.

Brouselle

Champagne

(2011). Program theory evaluation: Logic analysis. Evaluation and Program Planning, 34(1), 69–78. https://doi.org/10.1016/j.evalprogplan.2010.04.001

Brown

Hogg

Delva

Nanchoff-Glatt

Moore

(1999). Peer consultation reflection exercise. Innovative strategy for personal and professional development. Canadian Family Physician, 45(1999 Jun), 1509–1515.1038621510386215.

Chen

H.-T.

Rossi

P. H.

(1989). Issues in the theory-driven perspective. Evaluation and Program Planning, 12(4), 299–306. https://doi.org/10.1016/0149-7189(89)90046-3

Coghlan

(2019). Doing action research in your own organization. Sage Publications.

Coghlan

(2023). Action research. In Glăveanu

V. P.

(Ed.), The palgrave encyclopedia of the possible (pp. 9–16). Palgrave Macmillan. https://doi.org/10.1007/978-3-030-90913-0_180

Coleman

Ripping

(2000). Putting feminist theory to work: Collaboration as a means towards organizational change. Organization, 7(4), 573–587. https://doi.org/10.1177/135050840074004

10.

Coughlan

Coghlan

(2002). Action research for operations management. International Journal of Operations & Production Management, 22(2), 220–240. https://doi.org/10.1108/01443570210417515

11.

Dahmen-Adkins

Peterson

(2021). Micro change agents for gender equality: Transforming European research performing organizations. Frontiers in sociology, 6, Article 741886. https://doi.org/10.3389/fsoc.2021.741886

12.

Dahmen-Adkins

Peterson

(2019). Most significant change: Closing the gender gap in research. In Paolani

Paolani

Arduini

(Eds.), Proceedings of the 2nd international conference on gender research, ICGR 2019. Roma, Italy, 11 - 12 April 2019. (pp. 151–158).

13.

Dart

Davies

(2003). A dialogical, story-based evaluation tool: The most significant change technique. American Journal of Evaluation, 24(2), 137–155. https://doi.org/10.1177/109821400302400202

14.

Daudelin

M. W.

(1996). Learning from experience through reflection. Organizational Dynamics, 24(3), 36–48. https://doi.org/10.1016/s0090-2616(96)90004-2

15.

Dewey

(1910[1997]). How we think. Dower Publications, Inc. Originally Published by D.C. Heath & Co.

16.

Donnelly

Morton

(2019). Creating organisational and practice change through the use of co-operative inquiry groups in healthcare settings. Action Research, 17(4), 451–468. https://doi.org/10.1177/1476750319855126

17.

Espinosa

(2013). Moving towards gender-sensitive evaluation? Practices and challenges in international-development evaluation. Evaluation, 19(2), 171–182. https://doi.org/10.1177/1356389013485195

18.

Espinosa

Bustelo

Velasco

(2016). Evaluating gender structural change. Guidelines for evaluating gender equality action plans. Available online: https://hdl.handle.net/20.500.14352/27621

19.

European Commission . (2012). ROM handbook: Results-oriented monitoring. Brussels.

20.

Friedman

Rothman

(2015). Action evaluation: An action research practice for the participative definition, monitoring, and assessment of success in social innovation and conflict engagement. In The Sage handbook of action research (3rd ed., pp. 90–99). Sage Publications Ltd. https://doi.org/10.4135/9781473921290

21.

Frisby

Maguire

Reid

(2009). The ‘f’ word has everything to do with it. How feminist theories inform action research. Action Research, 7(1), 13–29. https://doi.org/10.1177/1476750308099595

22.

Funnell

S. C.

Rogers

P. J.

(2011). Purposeful Program Theory. Effective use of theories of change and logic models. John Wiley & Sons.

23.

Gatenby

Humphries

(2000). Feminist participatory action research. Women's Studies International Forum, 23(1), 89–105. https://doi.org/10.1016/s0277-5395(99)00095-3

24.

Haylock

Miller

(2016). Merging developmental and feminist evaluation to monitor and evaluate transformative social change. American Journal of Evaluation, 37(1), 63–79. https://doi.org/10.1177/1098214015578731

25.

Holbrook

J. B.

Frodeman

(2011). Peer review and the ex ante assessment of societal impacts. Research Evaluation, 20(3), 239–246. https://doi.org/10.3152/095820211x12941371876788

26.

Høyrup

(2004). Reflection as a core process in organisational learning. Journal of Workplace Learning, 16(8), 442–454.

27.

Kalpazidou Schmidt

Cacace

(2018). Setting up a dynamic framework to activate gender equality structural transformation in research organizations. Science and Public Policy, 46(3), 321–338. https://doi.org/10.1093/scipol/scy059

28.

Kotter

(1995). Why transformation efforts fail. Harvard Business Review, 73(2), 59–67.

29.

Kusek

J. Z.

Rist

R. C.

(2004). Ten steps to a results-based monitoring and evaluation system: A handbook for development practitioners. The International Bank for Reconstruction and Development/World Bank Publications.

30.

Lipinsky

Schäfer

(2016). INTEGER guidelines for the self-assessment of transformational gender action plans set up in higher education and research institutions. GESIS Leibniz Institute for the Social Sciences.

31.

MacArthur

Carrard

Davila

Grant

Megaw

Willetts

Winterford

(2022). Gender-transformative approaches in international development: A brief history and five uniting principles. Women's Studies International Forum, 95(November-December 2022), Article 102635. https://doi.org/10.1016/j.wsif.2022.102635

32.

Maestrini

Luzzini

Shani

Canterino

(2016). The action research cycle reloaded: Conducting action research across buyer-supplier relationships. Journal of Purchasing and Supply Management, 22(4), 289–298. https://doi.org/10.1016/j.pursup.2016.06.002

33.

McNiff

(2016). You and your action research project. Routledge.

34.

McNiff

Whitehead

(2005). All you need to know about action research. Sage.

35.

Mezirow

(1998). On critical reflection. Adult Education Quarterly, 48(3), 185–198. https://doi.org/10.1177/074171369804800305

36.

Neumann

Robson

Sloan

(2018). Monitoring and evaluation of strategic change programme implementation—lessons from a case analysis. Evaluation and Program Planning, 66, 120–132. https://doi.org/10.1016/j.evalprogplan.2017.09.012

37.

OECD . (2002). Glossary of key terms in evaluation and results-based management. OECD (Organisation for Economic Co-operation and Development.

38.

O’Neil

Marsick

V. J.

(1994). Becoming critically reflective through action reflection learning™. New Directions for Adult and Continuing Education, 1994(63), 17–30. https://doi.org/10.1002/ace.36719946304

39.

Patton

M. Q.

(2011). Developmental evaluation: Applying complexity concepts to enhance innovation. The Guilford Press.

40.

Pawson

Tilley

(1997). Realistic evaluation. Sage.

41.

Pawson

Tilley

(2001). Realistic evaluation bloodlines. American Journal of Evaluation, 22(3), 317–324. https://doi.org/10.1177/109821400102200305

42.

Podems

D. R.

(2010). Feminist evaluation and gender approaches: There’s a difference? Journal of MultiDisciplinary Evaluation, 6(14), 1–17. https://doi.org/10.56645/jmde.v6i14.199

43.

Pope

E. M.

(2020). From participants to Co-researchers: Methodological alterations to a qualitative case study. Qualitative Report, 25(10), 3749–3761. https://doi.org/10.46743/2160-3715/2020.4394

44.

Reason

Bradbury

(2008). The Sage handbook of action research: Participative inquiry and practice (2nd ed.). Sage.

45.

Reid

Tom

Frisby

(2006). Finding the ‘action’ in feminist participatory action research. Action Research, 4(3), 315–332. https://doi.org/10.1177/1476750306066804

46.

Riedy

Parenti

Childers-McKee

Teehankee

(2023). Action research pedagogy in educational institutions: Emancipatory, relational, critical and contextual. Action Research, 21(1), 3–8. https://doi.org/10.1177/14767503221150337

47.

Rimanoczy

Brown

(2008). Bringing action reflection learning into action learning. Action Learning: Research and Practice, 5(2), 185–192. https://doi.org/10.1080/14767330802185889

48.

Rossi

P. H.

Freeman

H. E.

Lipsey

M. W.

(2004). Evaluation. A systematic approach. Sage Publications.

49.

Schön

(1983). The reflective practitioner. How professionals think in action. Jossey-Bass.

50.

Scriven

(2013). Key evaluation checklist. Claremont Graduate University & The Evaluation Center, Western Michigan University.

51.

Sridharan

Nakaima

(2011). Ten steps to making evaluation matter. Evaluation and Program Planning, 34(2), 135–146. https://doi.org/10.1016/j.evalprogplan.2010.09.003

52.

Thaler

(2016). Learning organisations in science and research: The role of transfer agents in gender equality change processes. IFZ, University of Klagenfurt.

53.

Tosey

Visser

Saunders

M. N.

(2012). The origins and conceptualizations of ‘triple-loop’ learning: A critical review. Management Learning, 43(3), 291–307. https://doi.org/10.1177/1350507611426239

54.

UNDP . (2009). Handbook on planning, monitoring and evaluating for development results. United Nations Development Programme.

55.

van den Brink

Benschop

(2012). Slaying the seven-headed dragon: The quest for gender change in academia. Gender, Work and Organization, 19(1), 71–92. https://doi.org/10.1111/j.1468-0432.2011.00566.x

56.

Vaughn

L. M.

Jacquez

(2020). Participatory research methods – choice points in the research process. Journal of Participatory Research Methods, 1(1), Article 13244. https://doi.org/10.35844/001c.13244