Abstract
Introduction
The Theory of Change (ToC) is a recognized approach in transdisciplinarity to facilitate transformative learning processes. When coupled with a Serious Games (SGs) structure, effective tools for navigating sustainability transitions can be develop. This can be the case for the Theory of Change Game (ToCG) that we use in a particular real-word context.
Intervention
This study aimed to track the impact of ToCG versus other tools. A workshop was designed with 23 participants to plan the future of Tertúlias do Montado, a long-term initiative focused on the sustainability of the Montado silvo-pastoral system.
Method
In this experimental setup, participants alternated between the ToCG and two other tools. Discussion content and participant preferences were analyzed, allowing for a comparison between ToCG-based tools and those without ToCG principles.
Results
Results indicate that the ToCG facilitated an in-depth discussion where the societal problem of Montado’s decline, and the initiative’s primary goal were discussed. In the remaining tools these aspects were taken from granted without reflective examination. Additionally, the circular structure of the ToCG encouraged comprehensive discourse, steering participants away from simplistic or immediate solutions to complex problems. All tools received positive evaluations while the ToCG was particularly noted for introducing a novel concept and enabling visual exploration of diverse perspectives. Some participants, however, found its rules ambiguous and noted monotony after multiple rounds. To the best of our knowledge, this experimental design is new as well as its application in a real-world context.
Discussion and Conclusion
The distinctiveness of ToCG was evident in the quality of discussions and the resources required for its implementation, especially facilitator training. In contrast, the other tools required minimal preparation and could be applied without specialized facilitation. Therefore, ToCG is particularly valuable in contexts where deep reflection is essential, whereas simpler tools may sufficient in more operational settings.
Introduction and Background
The essence of transdisciplinary research (TDR) fundamentally relies on a holistic approach to address complex issues, leveraging methods and expertise to drive transformative change. This approach involves collaboration between diverse academic disciplines and non-academic partners to generate insights and actions aligned with broader research impact strategies (Armitage et al., 2019). An increasingly used structured approach within TDR is the Theory of Change (ToC), which outlines the causal logic behind interventions, enabling impact planning in a testable manner (Claus et al., 2023; Reinholz & Andrews, 2020).
The Theory of Change (ToC) is a comprehensive methodology for planning, implementing, and evaluating complex initiatives by mapping out the change process step-by-step (J. P. Connell & Kubisch, 1998; Vogel, 2012; Weiss, 1995). It articulates the relationship between long-term goals and the intermediate steps required to achieve them, identifying assumptions and external conditions that may influence outcomes. This approach is widely applied in fields such as social, environmental, and educational projects to clarify pathways to impact, monitor progress, and adjust as necessary.
ToC offers a more sophisticated approach by requiring a detailed explanation of how a program is expected to work and the underlying assumptions, which allows a clearer understanding of how interventions lead to outcomes, making evaluations more structured and adaptable as initiatives develops (Wilkinson et al., 2021).
ToC offers a sophisticated approach by requiring detailed explanations of how a program is expected to work, as well as the underlying assumptions, facilitating a clearer understanding of how interventions lead to outcomes. This makes evaluations more structured and adaptable as initiatives develop (Wilkinson et al., 2021). Consequently, ToC has gained prominence as an alternative to traditional logical planning frameworks, providing adaptable tools for monitoring and evaluation in uncertain project environments (Archibald et al., 2016). This model emphasizes learning in complex problem contexts, guided by those directly affected, while adopting a systemic perspective that links outcomes to diverse factors and connections at various scales (Armitage et al., 2019). Unlike conventional tools, ToC offers a more transparent approach for navigating complex change processes, aligning better with the unpredictable nature of outcomes in diverse contexts (Archibald et al., 2016). It is seen as a strategic tool for visioning, planning, communication, and evaluation, mapping actions towards long-term objectives and fostering reflection on mechanisms driving change in specific contexts (Deutsch et al., 2021; Reed et al., 2023).
According to Claus et al. (2023), a well-structured ToC process facilitates a shared understanding among stakeholders, enables contextual analysis, encourages brainstorming on project contributions to change, supports capacity-building, and guides decision-making and learning processes. However, despite its recognized value, the lack of consensus on the definition of ToC poses challenges, hindering collective improvement of ToC modeling theory and creating unrealistic expectations about its capabilities (Archibald et al., 2016).
Nevertheless, ToC remains promising for future research by involving stakeholders, fostering a shared vision, and highlighting the importance of adaptability. It offers a framework to uncover common assumptions about change across disciplines and demonstrate potential outcomes and impacts for research funders (Reinholz & Andrews, 2020). Combining ToC with other tools aids in planning for societal impact resulting from research interventions (Deutsch et al., 2021; Reed et al., 2023). To ensure that ToC is applied collaboratively and iteratively, its combination with Serious Games (SGs) is promising.
SGs have been gaining prominence in the literature as useful tools for addressing complex problems and facilitating learning in various domains such as economics, social policies and environmental management (Edwards et al., 2019; Stanitsas et al., 2019). With board and computer games being the most prevalent types, these games take advantage of the entertainment features to engage users in achieving specific goals in real-world contexts (Stanitsas et al., 2019). According to Page et al. (2016), playfulness is instrumental in stimulating creativity, transitioning the brain from a cognitive, rule-based state to a more relaxed state where the whole body is engaged in problem-solving (Page et al., 2016).
Edwards et al. (2019) see games as social learning platforms, that allow participants to immerse themselves in long-term problem solving (Edwards et al., 2019).These characteristics make SGs an increasingly valued tool in education due to the positive effects on student engagement, motivation, and learning across different educational levels (Garcia et al., 2016; Noroozi et al., 2020).Research has followed this recognition of SGs in learning contexts, fostering the development of a significant field of investigation within the education arena(Kara, 2021; Ullah et al., 2022).
The use of Serious Games (SGs) for professional audiences is also growing, particularly focusing on sustainability agendas and social learning, serving as effective tools to encourage knowledge acquisition and promote decentralized collaboration. The exchange of perspectives is facilitated by SGs, fostering dialogue and the construction of shared visions.
As research tools, SGs enhance awareness and literacy on specific topics, encourage inclusive discussions, and increase participant engagement (Dernat et al., 2022). By fostering a safe environment where players make decisions without real consequences, SGs enable immersive simulations that develop skills necessary to face complex challenges. Within these simulations, participants actively collaborate to create imagined futures, guiding their in-game actions and contributing to a collective vision of the future (Ampatzidou et al., 2018; Rodela & Speelman, 2023; Vervoort et al., 2022).
These interactions foster cognitive, social, emotional, and relational skills, making SGs effective in catalyzing change (Rodela & Speelman, 2023). Furthermore, the retention of skills and knowledge gained during games has long-term benefits, positively influencing motivation to collaborate and enhancing recognition of other participants’ contributions as valuable resources (Dernat et al., 2022).
But despite the adaptability of SGs to various contexts and their potential for transformative learning processes, Tribaldos and Schneider (2021) emphasise the rarity of transdisciplinary (TD) approaches within gaming.
The advantage outlined underpin the interest of increasing the use of SG in TD collaborations as they can be an effective tool to understand dynamic social-ecological systems and interventions aiming to induce real-life changes, accommodating participants with varying educational backgrounds and levels of engagement (Rodela & Speelman, 2023; Tribaldos & Schneider, 2021).
Given the above, SGs can be a useful tool for structuring a ToC process and contribute to the best practices identified by Claus et al. (2023); Wilkinson et al. (2021), including facilitating a shared understanding among interested parties, supporting capacity building and guiding decision-making and learning processes, involving diverse perspectives and roles, in the planning, implementation or evaluation of projects(Claus et al., 2023; Wilkinson et al., 2021).
In this context and as far as we were able to determine, the first SG that brings ToC thinking into TD research projects is described by Tribaldos and Schneider (2021).
The Theory of Change Game (ToCG) was designed to enhance practices in TD sustainable development (SD) projects through a systematic process encouraging ToC thinking and fostering collaborative discussions. The game serves the dual purpose of inducing strategic thinking that can promote changes while designing new projects and or evaluating existing ones, encouraging participants to reflect on their approaches and assumptions as they immersed themselves in the gameplay. Developed through iterative conceptualization, formalization, and testing, the ToCG included several testing sessions and events involving societal actors (Tribaldos & Schneider, 2021). This first application of the ToCG provided a platform for addressing crucial questions in sustainability projects, prompting participants to critically assess their assumptions and methodologies (Tribaldos & Schneider, 2021).
Considering the interest of this level of discussion and following the approach used by the authors we developed a ToCG to our case study. Our goal was not only to take advantage of the benefits of the game but also to implement an experimental design to test its capacity and identify the difference between the use of the game and other planning tools.
By comparing the outcomes achieved through the ToCG tool with those derived from non-gamified tools, the study discusses the cost and benefits that ToC can imply in planning tools within a TDR context.
To the best of our knowledge, this experimental design has not previously been documented. This paper provides a detailed description of the methodology used, including how data was collected and analysed. We moved forward with the description of results and discussion and conclude with the difference between tools, limitations of the experimental design and future research needs.
Despite the outlined limitations, this study provides a needed assessment of tools that can be used of TDR and hopefully inspires other colleagues to take advantage of real-world context to collect data on the difference of the discussion content considering different tools with similar objectives such as strategic planning.
Methodology
The Case Study: Tertúlias do Montado: A Dialogue Platform
The Tertúlias do Montado is an on-going initiative that started in 2016 with the goal of promoting a long-term TD dialogue platform addressing the contemporary decline of the Montado agro-silvo-pastoral system (Guimarães et al., 2024).
This system, covering approximately one million hectares in Portugal’s Mediterranean landscapes, is recognized for its high nature value and includes various Natura 2000 priority habitats. The Montado system integrates forestry, pasture, and animal production in a productive model that faces ongoing decline, presenting a complex challenge for which no simple solutions are available (Guimarães et al., 2018; Pinto-Correia et al., 2022).
Addressing the decline of the Montado, Tertúlias do Montado emerges as a multifaceted real-life situation that serve as an opportunity for testing and refining a TD approach (Guimarães et al., 2024).
The Tertúlias do Montado are founded on the premise that effective collective actions require long-term TD dialogue, fostering socialization between science and society and emphasizing the value of expertise in TD frameworks. As a dedicated platform for Problem Framing (Guimarães et al., 2024), this initiative enables collaboration among researchers from various disciplines or interdisciplinary fields and professionals from different sectors to define and enhance sustainability challenges related to the Montado agro-silvo-pastoral system (Figure 1). “Tertúlias do Montado” as a permanent problem framing Platform. In Guimarães et al. (2024).
In the first session, in April 2016, participants collectively developed a “common agenda” which remain continuously open to revision but that included 17 issues considered important to be collectively discussed. By October 2024, a total. of 40 indoor and outdoor sessions had been conducted, each averaging 30 participants.
In October 2022, a one-day workshop was organized to collective plan the future of Tertúlias do Montado. The event began with a presentation of findings from the first evaluation assessment, conducted four years after the initiative’s launch (Guimarães et al., 2024). These results served as a motto to evaluate the progress of the initiative (achievements, strengths, weaknesses), and reflect on the knowledge collected for strategic planning and future activities.
The Experimental Design of the Workshop
The workshop took place over a full day, with a lunch break, in October 2022. Combining the objective of this workshop with the relevance of testing tools that can be useful to TDR research, we developed an experimental design. To explore the specific characteristics and impact of using the ToCG, we designed an experimental framework that involved the use of ToCG and two additional tools. These tools were selected based on their ability to facilitate strategic planning for the future of Tertúlias Montadas.
ToCG tool was grounded in the ToC methodology and inspired by the one proposed by Tribaldos and Schneider (2021). While the other two tools deliberately excluded any gamification elements or ToG principles.
The three tools were: 1. An adaptation of ToCG 2. The revision of the Common Agenda that guide the development of the initiative 3. The development of an idea proposed by the participants as if they were the coordinators of the initiative.
A total of 23 participants participated in this workshop: 14 landowners and land managers, five researchers, and five representatives from public administration.
The workshop was designed to allow participants to engage with all three tools. To this end and considering the specific requirements of each tool, participants were strategically divided into rotating work groups, allowing them to interact with each of the three tools in sequence. This configuration facilitated a comparative analysis of group dynamics, allowing the assessment of potential variations in discussion results between tools. • In the first round, Groups A and B worked with the ToCG for 90 minutes. Meanwhile,
Groups C and D focused on the Revision of the Agenda and in Groups E and F assume the role of coordinators of the initiative, with these groups working for 45 minutes before switching. • In the second round, participants of Groups C and D and Groups E and F switched while Groups A and B continued working with ToCG. • In the third round, the participants of Groups A and B were divided to transition too other four groups. Participants from Groups C, D, E, and F were reorganized into two new groups to work with the ToCG.
This allowed for simultaneous play of the ToCG by two groups while the remaining groups were discussing the same topics using the two other tools. All tables included a facilitator, and one skilled facilitator coordinated the overall workshop. In total, we collected discussion content from four ToCGs and six from each of the other two tools.
The dataset included not only a content analysis of the discussion within the three tools, but also a questionnaire that assesses participants’ preference and opinion about the tools used and their experience in this workshop.
Description of the Three Tools Used during the Workshop
The adaptation of the Theory of Change Game
Considering the work developed by Tribaldos and Schneider (2021) (Figure 2) we developed a tailored game to encourage ToC thinking and collaborative discussions about the future of Tertúlias do Montado initiative. Board Game developed by Tribaldos and Schneider (2021).
The ToC was integrated into the game to facilitate critical reflection on the initiative’s underlying assumptions, aligning its objectives with the identification of potential actions and expected outcomes, while mapping the initiative’s impact as dialogue platform for supporting the preservation the Montado agro-silvopastoral system.
The game consisted of a board (Figure 3) and cards (Table 1). In the centre of the board the context is represented, in this case the context the initiative: Tertúlias do Montado (Figure 3). Based on Tribaldos and Schneider (2021) we included five main categories of discussion: The problem: which in this case was already defined as the current decline of the Montado in tree cover; The objective: which was also predefine as the maintenance of Tertúlias do Montado as a dialogue platform between those involved in the problem; The activities: to be developed in the future with an overall question of what activities should be developed; The results: what results does the initiative aims to achieve in the future; The connections: promoting a discussion between the relationship between the proposed activities and the overall objective of the initiative. Game board. The Questions Included in the Cards of the Game.
In each of these categories’ participants could play a card as described in Table 1. Each card included questions and instructions. The ToC principles are included into the game by means of the questions formulated (Table 1). Considering the difference between Tribaldos and Schneider (2021) context and the Tertulias do Montado initiative some questions were not included.
Before starting the game, a facilitator made a 10-minute introduction to explain the objective and rules of the game and gave each player an identifying pawn and five green cards, two blue cards, one yellow card and one red card. Additional cards of each category were kept in the deck, allowing players to draw them as needed during gameplay.
Following the framework established by Tribaldos and Schneider (2021), the content of the cards was developed to support game interactions. The green cards contained questions to introduce unexplored topics and to deepen the discussion, with some cards numbered to correlate with specific categories on the game board, prompting players to answer in relation to that category. The blue cards focus on the barriers and resources needed for initiative, the joker was a yellow card, while the red card enabled participants to progress to the subsequent topics.
The game was structured into two rounds. The first round served as an introductory phase designed to surface the primary discussion topics related to Tertúlias do Montado (Figure 3). In this round, each player rolled a die and contributed to the corresponding category on the board based on the number rolled. Since the board included only five categories, players who rolled a six had the option to roll again. Once all categories were addressed, the game advanced to the second round, which involved a more in-depth exploration of each category through targeted question cards. Players rolled the die again to determine which category they would address according to the cards in their possession. Players also had the option to exchange cards with the remaining deck on the table if needed.
To manage speaking time and ensure equal participation, a two-minute hourglass was used to limit each player’s speaking time to two minutes per turn. During each turn, participants recorded their contributions on post-its, noting the responses to the questions associated with each category and the card selected.
The board, dices, cards with the questions were the gamification features of this ToCG (Figure 4). Game elements and players.
The Common Agenda discussion
In line with the overall objective of the workshop, the Common Agenda discussion was one of the tools used to promote the discussion of the future of the initiative.
In the inaugural session of the Tertúlias do Montado in April 2016, participants collaboratively formulated what is referred to as the “Common Agenda”, which encompasses 17 issues that should be address collectively and profiting from the platform that Tertúlias do Montado was stabilising.
For the discussion of the Common Agenda, the tool was designed through four guiding questions: • what other topics should be included in the Common Agenda? • what topic(s) makes sense to discuss/work on again? • What are the priorities? • What other activities haven´t been developed so far and should be?
At the end of the discussion, participants ranked the topics in order of priority. They then explored the objectives for addressing each topic, discussed how the analysis of each should be structured, and identified key stakeholders to invite.
In the Role of the Coordinator
This tool was designed to allow participants to incorporate the role of coordinator of the initiative and from this standing point reflect on what they would do if they were the coordinators of the Tertúlias do Montado. To guide discussion, each participant was given a sheet with the following questions: 1. If you were the coordinator of the Tertúlias do Montado, what would you propose to do (types of activity)? 2. How would you implement it? 3. What limitations do you identify in the implementation? 4. What can fail? 5. How to resolve the identified limitations? 6. Suggestions
Following the underlying logic of this approach (putting yourself in the position of the other) and creating interaction in the group, the process begun with each participant answering the first question and then passing their sheet to the person next to them, who used the previous answer to respond to the second question. This rotation of sheets continued, ensuring that each successive question was addressed by a different individual, until each participant received their original sheet again, completing the cycle of sharing ideas.
Collecting the Perception of Participants
The results of all the tools were recorded in sheets used during the groups discussions. Additionally, each facilitator recorded their observations and insights from the discussions they led and finally in a plenary setting all groups provided a brief presentation of the main discussion points generated while they engaged with the different tools. This plenary session was audio-recorded for further analysis.
At the end of the session an evaluation was carried out to assess the participants’ perception about the session. Two questionnaires were distributed for this purpose: one focused on evaluating the tools and the other the overall session.
In the tool evaluation questionnaire, participants were required to identify the tools they interacted with and rate their satisfaction on a scale from 1 (very dissatisfied) to 5 (very satisfied). They were also encouraged to provide detailed feedback on their satisfaction levels, explaining what they liked and disliked and explain the reasons for their responses.
The second questionnaire focused on the overall dynamics of the session. Participants were asked to rate their satisfaction with the meeting’s structure, including the sequence of themes, presentation, and overall dynamics using a scale from (very dissatisfied) to 5 (very satisfied). Lastly, participants were asked to assess the degree of innovation of this approach on a scale from 1 (not at all innovative) to 5 (Very innovative), specifically concerning methodologies, structure, and themes presented.
Data Analysis
To understand the difference between the three tools, a content analysis was conducted on the results obtained from each tool and the responses gathered through the questionnaire. All contributions were transcribed and organized by typologies or primary themes, allowing for both qualitative and quantitative analysis. During the content analysis, attention was given to identifying discussion points that were common across all tools as well as those unique to specific tools.
Results
This section is divided into two set of results. First, we compare the content of discussion to understand if the content between tools differs, secondly, we analyse the feedback retrieve with the questionnaires.
Differences and Similarities Between the Discussion Content of the Three Tools
Only in the case of the ToCG tool, participants questioned the primary objective of the initiative, agreeing that the core element, which should be preserved, is the co-creation of knowledge grounded in participants’ experiences and the integration of available scientific knowledge. Quantified results considering all the four times the game was play included: • Eight references to the goal of knowledge creation and sharing as a future objective of the initiative, • Ten references to the platform’s potential to promote behaviour change among producers and influence policy formulation, • Twenty-four references emphasizing the value of this format in strengthening the capacity of those addressing challenges within the Montado system.
Specific new activities were identified such as organizing expert-led sessions to present and analyze economic, environmental, and social outcomes within the Montado sector, as well as of case studies showcasing both successful and problematic management practices.
In the case of ToCG participants explored other organizational structures that should be linked to Tertúlias do Montadoo that at that moment served mainly as a dialogue platform. Participants envisioned new activities such as organizing expert-led sessions to present and analyse economic, environmental, and social outcomes within the Montado sector, as well as of case studies showcasing both successful and problematic management practices. The proposals also included practical and technical training with visits to scientific institutions and laboratory experiments, and the establishment of experimental real farms setting to co-create and test new farming practices, integrating researchers and framers in a community of practice. This aspect, unique to the ToCG tool, was not reflected in discussions facilitated by other tools. Furthermore, the ToCG generated seven references reflecting on academia’s role in knowledge creation and dissemination, with suggestions that the Montado sector itself might fund scientific research and the Tertúlias do Montado initiative.
As expected, in the Common Agenda tool, most of the discussion focused on specific themes to be addressed in future sessions of the initiative. However, participants were also encouraged to discuss other possible activities that could be developed by the initiative, yet in all six sets of results the groups consistently did not identify any other activity and focused on exploring themes in the common agenda. Participants prioritized revisiting “The life in the soil and its importance in the sustainability of the system” (10 votes). New suggested themes included raising awareness among younger generations and future successors of Montado land (most voted), soil microbial activity, and the carbon balance in the Montado system. The unique feature of this tool was the concrete identification of new topics and in-depth discussion on how these topics should be addressed.
In the tool ‘In the role of the coordinator’, new activities were identified and the discussions emphasized achieving tangible session outcomes while recognizing some topics may need multiple sessions. The main reflection point within this tool was how to arrive at more tangible results and how to improve communication and dissemination of the initiative. In four of the six groups that engaged with the tool communication plans were proposed. Funding was also a major concern, and participants suggested the application for financial support for a more permanent structure and result dissemination.
Across the three tools, several common themes emerged. Notably, references to field visits, workshops, and training were prevalent, with 13 proposals identified in the Theory of Change Game (ToCG), four in the Common Agenda tool, and 10 in the “In the Role of the Coordinator” tool.
A recurring topic was the importance of participant diversity within the initiative. Each tool highlighted the need to involve younger generations, with suggestions for targeted events aimed at engaging youth. Additionally, participants across all tools emphasized the importance of securing regular involvement from various entities, including public bodies, local stakeholders, and representatives from all levels of government.
There was also a consistent call across the tools for the involvement of decision-makers, underlined by the rationale that effectively communicating stakeholders’ concerns to those with policy influence is essential for driving transformation within the Montado system.
There were also common features not between all but on both ToCG and in ‘In the role of the coordinator’ tools, which was the discussion of issues around financial and human resources needed to secure the platform activities and potential to increase its capacity and diversity of activities.
Assessment of the Tools by the Participants
Regarding the evaluation of the tools used in the session, the results show that of the 23 participants who answered the questionnaire: 17 used all the tools, three only the ToCG, two the ToCG and ‘In the role of the coordinator’, and one the Common Agenda and ‘In the role of the coordinator’. Since, not all participants could follow all the session, they were integrated in the dynamics as they arrived.
The degree of satisfaction about the three tools was positive (between satisfied and very satisfied). Although no comparison between tools was made, in Figure 5 we can observe that the Common Agenda obtained the most positive evaluation with most participants very satisfied (47%) and satisfied (42%). Graphic on the degree of satisfaction of the participants regarding each tool used.
The tool where lower satisfaction levels were recorded was the ‘In the role of the coordinator’ that was the only one that obtained a less positive classification, with one participant not very satisfied.
The ToCG tool received 18 comments highlighting the positive aspects, and 16 comments addressed negative aspects. In summary the tool was described as providing a setting to reflection on the dynamics of the Tertúlias do Montado, facilitating open and informal debate on the different perspectives of the participants, allowing free circulation of ideas on different topics, enhanced by the interactivity of the game and the nature of the questions presented. Participants stated that the game enhanced personal interventions taking participants out of their comfort zone. As negative aspects, it was mentioned unclear rules and heavy reliance on moderators. Some participants reported difficulties in understanding the methodology, mentioning monotony and circularity due to excessive interconnection between themes and the lack of knowledge for informed decisions. Shortening the duration was suggested to mitigate this feeling of circularity.
The Common Agenda tool received 15 positive and five negative comments. The positive aspects included the ability to be retrospective and exploratory simultaneously and to lead to multidisciplinary thinking. As negative aspects, it was mentioned that it was somewhat confusing and that more time would be needed to deepen the discussion.
The ‘In the role of the coordinator’ tool received 14 positive and 5 negative comments. Positive aspects emphasized its clarity and the challenging to put yourself in someone else’s shoes and develop ideas from different perspectives from your own. As negatives aspects, it was mentioned that reading the other participants’ texts made the dynamics confusing.
Regarding the session, 20 participants replied to the survey and evaluation results indicate that a large majority expressed high satisfaction with the way the meeting was conducted: 35% satisfied and 65% very satisfied.
This session was considered by 60% of participants as innovative. All interviewees stated that the meeting’s structure served as an effective mean of communication among different stakeholders and an interesting approach for facilitating discussions and assessments of session performance.
Discussion
Did the Theory of Change Game Bring Something Different?
The interactive nature of ToCG helped facilitate discussions, challenging participants to step out of their comfort zone promoting personal interventions. The tool stood out by enhancing personal interventions and challenging participants to step out of their usual roles and think creatively, proving to be useful in creating an inclusive and welcoming dynamic. It diluted power relations, leading to broad participation (Hansson & Polk, 2018; Miller & Wyborn, 2020).
ToCG was the tool that provided a more explicit discussion about the role of Tertúlias in addressing the complex challenges that the Montado faces. Although participants did not explicitly reference the concept, their discussions aligned with and implicitly validated the TDR approach of the Tertúlias do Montado, as a transdisciplinary research process focused on societal problem solving, and as purpose-driven collaborative processes of knowledge production among re-searchers of different disciplines, inter- and trans-disciplinary fields, and communities of practice(Pohl et al., 2021). Feedback from participants echoed findings by Tribaldos and Schneider (2021), mainly the acquisition of knowledge about ToC thinking, understanding its potential, discussing the importance of incorporating and maintaining other societal actors engaged in the initiative, reflecting on their assumptions about contexts, and about the alignment of initiative activities with goals, while challenging their own views.
Limitations of the Experimental Design and Suggestions for Further Future Research
The experimental design in a real-world context presents challenges due to the need to balance data collection with the objectives that originally justified the participant gathering in the workshop setting. Several aspects which cannot be controlled, inevitably influence the outcomes. Nonetheless, conducting such experiments is a unique opportunity to test tools. Therefore, we encourage others to undertake similar experiments, as collecting data and discussing the effectiveness of existing tools are fundamental.
When participants are gathered with the explicit purpose of testing a tool, the results are inherently influenced by this objective. Although real-world contexts are often complex and unpredictable, they allow for the tool’s use to stimulate the type of discussion participants expect. Such settings provide important insights that need to be increasingly shared within the TDR community.
We acknowledge the limitations of assessing tools through a single case study. While outcomes may be influenced by the profiles of facilitators and participants, consistent patterns were still observed across working groups with different compositions and facilitators. However, a broader range of case studies would provide more robust insights across diverse contexts. Such variations are essential to understand whether trends and challenges with ToCG remain stable or vary according to contextual factors.
Still regarding the experimental design, our data collection was limited to two data sets: 1) the content of discussions and 2) survey responses. Incorporating participatory observation to formally assess learning outcomes could enhance the design, providing an additional data that could provide valuable insights.
As previous studies have shown, our results indicate that developing ToC is a challenging process that requires time for reflection, multiple attempts, and the significant involvement of researchers and stakeholders (Oberlack et al., 2019; Wilkinson et al., 2021). Adapting ToC for multi-actor settings through gamification presents a promising approach to enhancing engagement and accessibility (Claus et al., 2023; Rodela & Speelman, 2023; Sousa, 2021).
Our ToCG was played by farmers, public administrators and researchers, who reported high levels of satisfaction. The results indicate that the game format contributed significantly to this positive feedback, reinforcing the recommendation by Rodela and Speelman (2023) to use tools like SGs to enhance engagement in complex processes. As reported by Sousa (2021) we also experience challenges in game preparation, such as limited experience, scarce resources, and the need for adaptation to various contexts (Sousa, 2021).
Future empirical studies aimed at tool comparison should prioritize assessing the cognitive load experienced by participants. In this study, participants reported significant fatigue because of engaging with three distinct tools. Limiting the number of tools or rounds within ToCG could be an effective strategy for reducing this cognitive burden. Additionally, to prevent a sense of repetition and monotony, dividing related themes into shorter, more manageable segments may enhance engagement. The inclusion of visual aids and an introductory tutorial session is also recommended to improve comprehension and facilitate participant involvement (Sousa, 2021).
A possible way forward could also involve refining the ToCG with the input from participants to increase the clarity of the rules and make the game easier to understand. According to Sousa (2021), enhancing acceptance and relevance may be achieved by incorporating co-design and participatory design methods that engage participants directly, which can improve the integration and practical application of the games (Sousa, 2021). In this context, future research should continue to explore ToCG as a participatory tool, integrating insights from stakeholders and existing literature on SG and ToC. Collaborating with game designers, developers, and researchers can help creating effective SGs that enable iterative, collaborative application of ToC, supporting TDR and bridging the gaps between disciplines as well as between research, policy and practice.
While engaging in gameplay can facilitate a deeper understanding of various scenarios, it is essential to reflect on the insights gained and consider their practical applications. As suggested Tipton et al. (2016), genuine learning often commences once the game concludes. Within this context, debriefing is essential as games can trigger a wide spectrum of emotions, from satisfaction and accomplishment to frustration and cognitive dissonance (Crookall, 2014; Tipton et al., 2016).
Therefore, repeating the ToCG in the context of Tertúlias do Montado can assist in design, monitoring and evaluation, reinforcing the benefits of this approach for TDR. However, it is important not to perceive the ToCG as an end goal. Instead, view this tool as a means of promoting dialogue, facilitating knowledge sharing and achieving a broader purpose.
Conclusions
We successfully developed a Theory of Change Game (ToCG) tailored to our specific objectives, contributing to the testing and evaluation of methodological tools. With the experimental design develop we were also able to assess the distinctive features of approach of ToCG. ToCG was found to effectively guided to the reflection that ToG aims to promote while the gamification facilitated the process. Despite the features of ToCG, all tools allowed planning in a multi-actor setting. Hence, although ToCG has distinct advantages, it should not be viewed as a universal solution, as its implementation requires additional resources and careful planning. Decisions to use this tool should account for the necessary allocation of resources for both design and implementation. Furthermore, TDR should encourage self-reflection processes to evaluate the effectiveness of such methodologies and tools, assessing their alignment in the pursuit of the objectives and the extent to which desired outcomes are achieved. Although the value and potential benefits of these tools are frequently discussed, studies that examine their practical application are lacking. We hope this study serves as a model for such research and motivates further contributions to this essential area of knowledge.
Footnotes
Acknowledgements
The authors would like to greatly thank the participants of Tertúlias do Montado, without their commitment and interest, this initiative would have never been possible. For language improvement, we used ChatGPT-3.5.
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This study received Portuguese national funds from FCT - Foundation for Science and Technology to MED trough Project UIDB/05183, MED (https://doi.org/10.54499/UIDB/05183/2020; https://doi.org/10.54499/UIDP/05183/2020); to CHANGE (https://doi.org/10.54499/LA/P/0121/2020).
