Sage Journals: Discover world-class research

Abstract

Introduction

The role of reflection in experience-based learning is discussed widely. This paper researches the effects of a reflection assignment as part of debriefing with qualitative and quantitative methodology in five simulation game-based seminars.

Research Design and Methods

The intervention (reflection assignment) used in this study consists of three reflective questions that are discussed in groups. We aim to find out the effects of adding this intervention to the regular in-between-debriefings (three times during the whole seminar). The methodological setup is quasi-experimental and so, consisted of a test group and a control group. Two different types of simulation games were used, one a general management game and the other a change management game. Data used in this study is gathered from three different sources: First, we used in-game-performance data from both games. Second, we used two types of questionnaires to evaluate game experience and self-reported learning (MEEGA+ and ZMS inventory). Third, the reflection notes from the treatment-teams were used for qualitative analysis.

Results

In several dimensions (game-success, game-experience and self-reported learning) significant differences were found between treatment and non-treatment students. Qualitative data show a deep level of reflection for treatment teams differentiating for the two simulation games evaluated.

Discussion

Supported by the qualitative reflection data we assume that the better results for treatment-students are routed in the repetitive reflection. The permanent circular reflection helps treatment students to understand the games better and gain deeper insights.

Keywords

reflection assignment debriefing in-between-debriefing learning intervention qualitative and quantitative research mixed-method research

Introduction

In this research the effects of a formative assessment method consisting of three reflective questions is evaluated using a mixed method research design. In five sg based seminars treatment groups received an additional reflection assignment while non-treatment groups had “only” the normal debriefing. Differences between treatment and non-treatment groups are evaluated in various ways and qualitative analysis shows how reflection adds value for learning. The sgs used in this study are TOPSIM General Management (GM) and Riva Sys Teams Change (STC). To introduce the theory behind our research we draw the connection between reflection and learning and describe why this connection is relevant for learning with sg. After introducing the theory on levels of learning we outline the research-design and detailed research questions (RQ) for this study. So then we describe in detail how each RQ is analyzed (methods). After presenting the results of this study per RQ we discuss our findings and name limitations.

Introduction to the role of reflection in learning

The idea of sgs as an experiential learning method is “to provide an enhanced learning experience” (Pasin & Giroux, 2011, p. 1243). A simulated learning environment gives students the chance to experience, reflect and learn. Usually, sgs follow the structure of briefing, experience and debriefing (Duke, 2014; Geurts, Duke, & Vermeulen, 2007; Klabbers, 2009; Stoppelenburg, Caluwé, & Geurts, 2012). Briefing serves to introduce students to the simulation and prepare them with what they need to know about the sg (scenario, rules, roles and resources etc.) to start playing it. Part of briefing can be establishing goals for learning which can contribute to reflection later on. While the sg is played students act in their roles and gather experience, which then serves as input for reflection and debriefing both during and after the sg (Schwägele, Zürn, Lukosch, & Freese, 2021).

In literature sgs are associated with experience- and action based learning theories (Kolb & Kolb, 2009; Taylor, Backlund, & Niklasson, 2012), for example Kolb’s Experiential Learning Cycle (Alklind Taylor, 2014; Geithner & Menzel, 2016). It describes learning as a continuing cycle of concrete experience, reflection, abstract conceptualization and active experimentation (Kolb & Kolb, 2009). Learning can occur when experience is reflected and conceptualized. The new concepts are transferred to action, which leads to new experiences. Argyris and Schön developed another systemic oriented experiential learning theory and identified the concept of first and second order learning (1974) which is described in detail below. Schön adds the concept of reflection-in-action and reflection-on-action (1983). He elaborates how professionals handle (challenging) situations based on their experience by immediate reflection during their action (reflection-in-action) and reflecting on it afterwards (reflection-on-action). Relating to the concept of Schön, Greenwood adds the idea of reflection-before-action, meaning to anticipate a situation beforehand (1993).

Many authors emphasize the relationship between experience and reflection in experiential learning theories (Alklind Taylor, 2014; Hilzensauer, 2008; Klabbers, 2009; Kolb & Kolb, 2009; Nakamura, 2022). In sum, sgs offer opportunities to learn from experience through reflection. In the game play phases students aim to solve simulated challenges and thereby gather experience as an input for reflection/debriefing.

Contemporary research on reflection in simulation games

Reflection has a role in the learning processes within sgs (Baker, Jensen, & Kolb, 1997; J. Y. Lee, Donkers, Jarodzka, Sellenraad, & van Merriënboer, 2020; Nakamura, 2022). Adding reflection assignments during the gameplay is a form of formative assessment which enables participants to use the learnings from reflection and utilize the sg to the full. Only reflecting in debriefing after gameplay would leave out many opportunities for learning, whereas adding reflection triggers and opportunities during a sg session can serve as leverage points for learning. Specific research on the different forms of reflection and their effects in sgs is scarce at the moment. Nakamura (2022) researched the impact of adding pre-structured questions before the gameplay and in reflections and found a positive significant result on learning. Lee et al. (2020) performed a medical simulation study in which participants could take time outs resulting in higher experienced agency but not in an increased learning outcome. The authors explain the result by concluding participants did not receive specific instruction or prepared questions on how to use this reflection time effectively. Husebø et al (2013) researched the effects of debriefing questions on reflection during debriefing after gameplay with participants in medical sgs and found a positive (qualitative) result and suggests further research is necessary on types of questions asked and their effects on learning. We do not know if these results are generalizable to sgs in general, since medical sgs often have specific contexts such as procedural orientation with given norms on what behavior is correct according to pre specified procedures. However, there are also medical sgs having more open characteristics for instance when aimed at more tacit knowledge and competencies such as situational awareness.

These findings indicate structured questions could add value to (in-between-) debriefing to increase learning, however the number of studies so far is limited. This study aims to contribute to researching the effects of structured reflection assignments on learning with sgs.

Theoretical background on levels of learning

Argyris and Schön identified the concept of first and second order learning (1974) which has been used and referenced widely in management literature (Bartunek, 2014; Crossan, 2003; Friedman, Torbert, Nielsen, Silverman, & Bradbury, 2014). The basis of systemic oriented learning is seen in reflecting on and responding to deviations that happen in a given system. The type of response determines the type of learning involved. Theory development from the founding father of systems learning Chris Argyris is applicable here. First order learning can be defined as learning to apply predetermined rules and procedures in a given system. It leads to successful actions within the system. Second order learning requires a reflection on the norms behind the decisions and assumptions so processes can be adapted in such a way they fit the requirements of the situation. These process interventions are directed at checking if the norms applied are still relevant and adequate for the challenges the people in the organization are faced with (Argyris, 1977; Greenwood, 1988; Tosey, Visser, & Saunders, 2012). Greenwood (1988) emphasizes that especially second order learning is based on reflection because it criticizes the underlying structures and norms behind them. The concept of third order learning was introduced based on the work of Argyris as an extension of his theory (Tosey et al., 2012; Visser, 2007; Visser, Chiva, & Tosey, 2018). Third order learning means to reflect on how actors take on their roles in the system and if this is adding value to their personal values/needs and the values/needs of the system. In addition, third order learning is about learning to learn and remain adaptive. In this process also less tangible issues regarding virtues of wisdom and justice can be involved (Reynolds, 2014). Applying these levels of learning to the qualitative part of this study (especially research question 3 see next paragraph) enables us to draw conclusions on what type of learning was most prevalent and if this was in line with the aim of the specific case.

Aim of research and research questions

The aim of this research is to evaluate the impact of a structured reflection assignment that is repeated three times during the sg-based seminars as a part of the roundly in-between-debriefings. Therefore, treatment groups participated in a reflection assignment while non-treatment groups did not. The groups are then compared in various aspects using quantitative and qualitative methods. In detail the following research questions (RQ) are addressed in this study:

- RQ1: How does the reflection intervention affect the sg results and perceived learning in the treatment groups?

- RQ2: How does the reflection intervention affect game play variables (assessment of in-between-debriefing and debriefing, student engagement, teamwork, assessment of simulation)?

- RQ3: What levels of learning/reflection do we see in the notes on the flips and can there be made interpretations on in what proportion they occur?

- RQ4: What learning process do we see in the teams from round to round?

Research design and methods

Overview research design

To address the formulated RQ a mixed method setup was used (see Figure 1). Quantitative evaluation was applied with two post game questionnaires and qualitative analysis was applied to the reflection notes gathered. Fife sg-based seminars were evaluated with a quasi-experimental design. Altogether 110 students participated in 25 teams. In each seminar two groups received a special reflection assignment with pre structured reflection questions (treatment groups) that was repeated three times during gameplay (beginning, middle, end). The other two to four groups (depending on how many students attended the seminar) did not receive the reflection assignment. In the treatment groups the assignment became part of the in-between-debriefing and was added after the discussion of results in the plenary. For non-treatment groups the in-between-debriefing ended after the plenary discussion. So, these groups had a little (about 10 to 15 minutes) more time to read, calculate and discuss their next round's decisions within their teams. The treatment was implemented three times during the sg-based seminars: At the start (after round one), in the middle of gameplay (round 3 for GM and round 2 for STC) and towards the end of the game (round 5 for GM, round 3 for STC). The procedure differentiated slightly because GM consists of six rounds and STC of three. There was no interaction between the groups at the reflection moments. The reflection-assignment exists of three pre structured questions that guide the teams to reflection on their decisions and behavior during the sg:

1. What went well?

2. What went not so well?

3. What do I need, what do we need to reach our goals?

Figure 1.

Overview research design.

These pre structured questions were open and quite general to allow participants the opportunity for personalized input on what they found relevant. Previous studies (J. Y. Lee et al., 2020; Nakamura, 2022) indicate that pre-structured questions support learning via reflection and also preparation with goal setting for the next game round.

In onsite seminars the teams received a prepared flipchart with invitation to discuss the questions written on the flipcharts and write their answers on the chart. In other case studies (Wijse-van Heeswijk et al., in press) we noticed participants were not so inclined to fill out the questions if they were not pre written on the flip over. In online seminars the teams received a prepared PowerPoint sheet with the same questions and were invited to discuss answers and make notes by screen-sharing. Afterwards the results were presented briefly to the professor and the other team.

At the start of the seminars students were informed that the seminar was evaluated in the context of a research project on the role of learning in sgs. Participation in the study was voluntary and data was gathered anonymously using Evasys Version 8.2.

Quantitative analysis approach (RQ1 and RQ2)

RQ1 focuses on the output of the seminar (learning and game results) while RQ2 focuses on the gameplay process (teamwork, engagement, assessment of simulation). To address these RQs students were asked to answer a longer questionnaire at the end of the seminars. These questions were aimed at reflecting on the seminar from start to end. The questionnaire consists of two inventories that are described below. Both inventories consider self-reported learning results (RQ1) as well as gameplay-process variables that are addressed in RQ2 (see Figure 1 for an overview). The first part of the questionnaire is called the ZMS inventory and was developed at the Center for Management Simulation. It measures learning satisfaction and relevant influencing factors on learning with simulation games (Trautwein & Alf, 2022) and consists of 27 Likert scaled items that can be combined to seven scales. Two scales measure the quality of the simulation game regarding comprehension of the game and the relation to realism of the game. One scale of five items measures how the simulation is facilitated. Two scales measure different aspects of teamwork (task orientation and atmosphere). Another scale measures the student engagement during the seminar and the final scale is about overall satisfaction and learning. Exploratory and confirmatory factor analysis confirmed the seven scales used in this research and alpha values between .783 and .943 can be reported for the scales (Trautwein & Alf, 2022). The second questionnaire is the MEEGA+ instrument developed by Petri et al. (2017) to evaluate educational games. The MEEGA+ questionnaire is a validated instrument and was originally developed for games with digital interfaces. Therefore, some questions were adapted to the games used in this study. For example, items used to evaluate the accessibility of games ask about fonts and colors. In the context of this study we asked “The game interface is clearly arranged.” and “The user interface is simple and well structured.” The social interaction scale of MEEGA+ asks if there was the possibility of interaction which is natural for the sg used in this research. So, we converted the question to whether the simulation promotes cooperation with other students. For quantitative analysis treatment students and non-treatment students are compared (for some analysis treatment groups and non-treatment groups are compared using group means). Since evaluation data on group level is n < 30 Mann-Whitney-U-tests are calculated for test statistics. U-Tests are appropriate for smaller samples which we have on group level and are even appropriate for non-parametric data (Bortz & Schuster, 2010; Eid, Gollwitzer, & Schmitt, 2011, p. 322).

To address RQ1 also success indicators from the sg are used knowing that this is controversially discussed (Brazhkin & Zimmerman, 2019; Pasin & Giroux, 2011). Critique on the use of game success indicators is that in some games groups could perform well by random luck instead of learning. We find this is very unlikely for the sgs used in this study because the simulations used have a high level of complexity. For GM students have to make many interdependent decisions in six consecutive rounds¹ which means that they have to bring their decisions into a coherent overall concept. Also, there is not one right solution but multiple strategies possible which have to be performed in a coherent way to achieve game success. For STC students have to play 42 action cards in a logical order according to the simulated change management process. For many action cards it is required to select proper game characters to play them which has a certain level of difficulty. Since many different characters in the game have to be approached it is highly unlikely one would select the right candidates by luck for each intervention every time. In both simulations it can be assumed as very unlikely that successful decisions can be taken by chance permanently. If a team performs well in the game, it indicates at least that they have learned to a certain extent how to play the sg in line with the learning goals of the sg.

Since RQ2 addresses process variables such as the quality of in-between-debriefings a short questionnaire was conducted after each in-between-debriefing. Students were asked to answer the perceived quality of debriefing using 3 items that were combined to a scale. Treatment groups filled it out after their assignment, control groups after the joint debriefing discussion. Via this procedure it is possible to compare the treatment groups with the non-treatment groups on the perceived quality of in-between-debriefing.

Qualitative analysis approach (RQ3 and RQ4)

In addition to quantitative data from questionnaires and game performance data the research design also provides qualitative data from the treatment groups. The notes from the reflection assignments were collected, documented, and used for qualitative analysis. This may grant insight into the level of group reflection and by comparing consecutive reflections of groups in learning processes. With RQ3 we look at the data on a micro level analyzing the single answers given by groups. With RQ4 we analyze the data in a broader view comparing the notes in a chronological manner.

With regard to RQ3 we converted the text from the flaps to an excel file and assigned codes according to the coding scheme in the appendix (Argyris, 1977; Visser et al., 2018): Code 1 for first order learning, Code 2 for second order learning and Code 3 for third order learning. Using these codes, we analyze the reflective response to find out what teams were involved with what learning levels during the seminar (these could either be actual learnings or reflections on certain learning levels). As described above, treatment teams participated in three reflection interventions during the process of gameplay answering three open questions (after round one, in the middle of the game and towards the end of the game).

Comparing the discussion-notes (flipcharts or digital notes) in a chronological manner gives insight into a process of discussion and learning the groups went through (RQ4). Applying the idea of process tracing (Cloutier & Langley, 2020; Collier, 2011; Langley, 1999; Langley & Tsoukas, 2016; Muno, 2016) we analyzed the data to find relevant chronological patterns. Therefore, five relevant scenarios of possible processes were formulated, marked with colors and the occurrence of scenarios was counted as is shown in the following table. (Table 1)

Table 1.

Description of the Five Possible Learning Scenarios.

	Short description	Description
Scenario 1	Topic occurs only in one period	Issue was identified in one period, but unclear how it was processed. This doesn’t necessarily mean participants did not learn how to deal with it but from the data we have we cannot deduct what happened to it.
Scenario 2	Positive → positive	Issue was identified as “worked well” in earlier rounds. Team managed the issue continuously in a satisfying way.
Scenario 3	negative → positive	Issue was identified as “worked not so well” or “What do we need to…”. Team managed to work on the issue so that it was identified as positive in later rounds.
Scenario 4	negative → negative	Issue was identified as “worked not so well” or “what do we need to…” in earlier rounds. Team was not able to work on the topic in a satisfying way so that it was identified as negative again.
Scenario 5	positive → negative	Issue was identified as “worked well” in earlier rounds. Team had problems dealing with the topic in later rounds.

Using the described scenarios, we see which topics were discussed when in the seminar and how the topics proceeded during the seminar. For example, scenario 3 negative -> positive shows that teams identified an issue as relevant but have so far not managed to work on the issue in a satisfying way (“went not so well” or “what do we need…”). In later rounds the topic is identified as “worked well”, indicating that teams found solutions to manage the issue properly. Especially scenario 3 can be interpreted as an indication for learning. By working on the issue it turned from negative to positive, indicating that students learned how to deal with the problem.²

Case study and context description

Using the described methodology we evaluated five simulation based seminars at the DHBW Stuttgart Business Faculty that were taught at the Center for Management Simulation (ZMS). The Center for Management Simulation is a specialized lab for sgs. The setup of the management simulation is as follows: In a plenary room the briefing, in-between-debriefing and final debriefing took place, during gameplay the teams have their own offices with a big screen to share data, whiteboards and flaps to share notes and organization. Due to the corona pandemic only two of the evaluated seminars were conducted on site (see Table 2). Three other seminars were conducted via MS Teams where teams had their virtual rooms. We evaluated four seminars using the simulation game TOPSIM General Management which is a complex business sg that has been used in scientific study before (Hühn & Rausch, 2022). Students take on the role of the board of a company and make decisions in six areas of management (TOPSIM GmbH). All teams start at the same position and compete with each other on one market. The most general performance indicator is the share price of a company. All four seminars were facilitated by an experienced professor using the simulation for approximately 18 years. In the fifth seminar a simulation on change management (riva solutions GmbH) was used. Students take over the role of external change consultants and guide a middle size organization through a change management process. The goal of the game is to move a representation of 26 characters from total rejection of the change-process to acceptance of the process. The simulation was facilitated by two experienced trainers that taught this specific simulation more than ten times. The actual plan was to evaluate at least two change management seminars. Due to the corona semester this was not possible. But it provides at least one example of another sg to discuss whether the effects are typical for GM or can also be generalized to other sgs. Both sg’s are round based simulations. After each round the simulations were “interrupted” for an in-between-debriefing discussion in plenary, that is led by the lecturers. Relevant topics and results of the last round were discussed and students had the opportunity to ask questions. After the in-between-debriefing in the plenary the treatment teams started their reflection assignment.

Table 2.

Overview of Seminars and sg Used for Evaluation.

Results/findings from both games

In this chapter the results of our research are described. In the text we follow the structure of the RQ raised above.

Figure 2.

game performance of treatment and control teams for GM.

RQ1: Effects on performance in the game and perceived learning

Comparing treatment teams and control teams for GM (n=19) we see basically no effects in the first rounds. Starting from round 5 we see treatment-teams having better game results (Figure 2). Results in round six show a mean share price of 410 for non-treatment teams and a mean share price of 485 for treatment teams.³

The game success measures for STC (n=6 teams) indicate that the two treatment-groups have the best results and the lowest standard deviation (Table 3). For STC the standard deviation is an important measure because it shows that a team was able to move all 26 characters simultaneously by their change management activities.

Table 3.

Game Success measures SysTeamsChange.

	Treatment	Treatment	Control	Control	Control	Control
Success (average of moved people on the game board)	21,27	21,42	16,62	18,46	20,39	19,5
Standard deviation	1,82	1,20	4,45	3,66	2,40	1,20

Looking at the results of both sgs, we see the tendency of better results for the treatment groups. Using z-standardization (Pospeschill, 2022, p. 190) the success measures of both sgs were brought together to one scale. Due to the small sample of team means (n=25) a Mann-Whitney-U-Test for independent samples was calculated to identify significant differences between reflection groups and non-reflection groups. The test shows a highly significant result (.008) and a medium to strong effect size of r=.521 (Table 4). It confirms what is seen looking at both games' individual success measures. There is a tendency of treatment groups having better results.

Table 4.

Results on Game Success and Learning (RQ1).

RQ1
Scale or Item		N	Mean Rank	Sum of Ranks	Mann-Whitney-U-Test	Sig. (2-tailed)	Effect size r
z-standardized success measures GM and STC	Control	15	9,87	148,00	28,000	0,008	0,521
	Treatment	10	17,70	177,00
	Total	25
Learning (MEEGA+)	Control	65	48,14	3129,00	984	0,004	0,278
	Treatment	45	66,13	2976,00
	Total	110
Learning (ZMS)	Control	65	51,55	3350,50	1205,5	0,117	0,149
	Treatment	45	61,21	2754,50
	Total	110
Learning: Interrelated business knowledge (single item MEEGA+)	Control	49	36,66	1796,50	571,5	0,002	0,334
	Treatment	37	52,55	1944,50
	Total	86
Learning: Operations management (single item MEEGA+)	Control	49	38,32	1877,50	652,5	0,021	0,249
	Treatment	37	50,36	1863,50
	Total	86
Learning: Cost calculation (single item MEEGA+)	Control	49	38,70	1896,50	671,5	0,033	0,230
	Treatment	37	49,85	1844,50
	Total	86
Learning: Marketing (single item MEEGA+)	Control	49	39,45	1933,00	708	0,075	0,192
	Treatment	37	48,86	1808,00
	Total	86

Interpreting students' perceived learning variables, a highly significant result is found on the MEEGA+ learning scale but not on the ZMS learning scale (Table 4). In detail three out of the four MEEGA+ learning single items show significant results: Treatment students assessed their learning better regarding cost calculation, operations management and for interrelated business knowledge (Table 4). The conflicting results between learning on MEEGA+ and the ZMS inventory may be due to the fact that the ZMS scale is formulated very general and consists of 6 items measuring learning and satisfaction on a general level (e.g. “I have learned a lot in the simulation game” or “How satisfied are you with the course overall?”), whereas the variables of MEEGA+ are formulated very specific⁴ and request students to assess their learning with regard to a very specific topic. From the results can be deduced that treatment groups and non-treatment groups assessed the sg not different concerning learning and satisfaction (ZMS inventory) but made a difference when they were asked for detailed learning goals.

RQ2: Effects on process and team variables

Comparing student’s assessment on round based in-between-debriefings we do not see a significant result when comparing treatment students with non-treatment students. We can state that the reflection assignment did not improve students’ perceived quality of the in-between-debriefings (Table 5). But we see a trend of treatment students to rate the debriefing better from round to round compared to non-treatment students (Table 5). While non-treatment students (control) rated the first in-between-debriefing slightly better, treatment students rated the second and third in-between-debriefing better with a tendency of greater differences towards the third measurement (end). This is also seen in the test-significance (becoming lower from time to time) and the effect size (becoming higher from time to time). Possibly the reflection assignment shows it’s effects only by repetition as we have also seen in RQ1. This interpretation must be made with caution because it might be due to coincidence as none of the tests show significant results.

Table 5.

Assessment of Debriefing with and Without Reflection Assignment (RQ2).

In-between-debriefing		N	Mean Rank	Sum of Ranks	Mann-Whitney-U-Test	Sig. (2-tailed)	Effect size r
Start	Control	51	44,99	2294,50	918,5	0,830	0,023
	Treatment	37	43,82	1621,50
	Total	88
Middle	Control	40	33,41	1336,50	516,5	0,318	0,119
	Treatment	30	38,28	1148,50
	Total	70
End	Control	43	35,56	1529,00	583	0,253	0,132
	Treatment	32	41,28	1321,00
	Total	75

Comparing sg process variables from the ZMS inventory such as student engagement or teamwork we see significant effects between treatment and non-treatment students with small and medium effects (Table 6). Treatment students evaluate themselves to be more engaged (Sig. .024, r=.216) and to have both, a better team atmosphere (Sig. .000; r=.351) and a better task orientation (sig. .020, r=.223). So, differences between treatment and non-treatment students are not only seen in results as described for RQ1 but also in process variables that possibly explain the different results. From logic and several research review articles on effects of agency and motivation (Bedwell, Pavlas, Heyne, Lazzara, & Salas, 2012; Deen, 2015; Watt & Smith, 2021), we may infer that teams show better results if students are more engaged and perform better as a team. Sg comprehension is exact on the .05-significance-level with a rather small effect size, meaning that treatment-students rate themselves having a slightly better understanding of the sg than non-treatment students. This is consistent with other results. Having a better understanding of the simulation may lead to better results and learning. No significant results are found for the scoring on the lecturers of the seminars and the practical relevance of the simulation.

Table 6.

Results for Process and Team Variables (RQ2).

RQ2
Scale		N	Mean Rank	Sum of Ranks	Mann-Whitney-U-Test	Sig. (2-tailed)	Effect size r
Student Engagement (ZMS)	Control	65	49,38	3210,00	1065	0,024	0,216
	Treatment	44	63,30	2785,00
	Total	109
Team_Atmosphere (ZMS)	Control	65	46,28	3008,50	863,5	0,000	0,351
	Treatment	45	68,81	3096,50
	Total	110
Team_Task Orientation (ZMS)	Control	65	49,62	3225,00	1080	0,020	0,223
	Treatment	45	64,00	2880,00
	Total	110
SG_Comprehension (ZMS)	Control	65	50,58	3287,50	1142,5	0,050	0,187
	Treatment	45	62,61	2817,50
	Total	110
Learnability (MEEGA+)	Control	65	48,99	3184,50	1039,5	0,009	0,248
	Treatment	45	64,90	2920,50
	Total	110
Social Interaction (MEEGA+)	Control	65	50,52	3284,00	1139	0,044	0,192
	Treatment	45	62,69	2821,00
	Total	110

Looking at the MEEGA+ scales the results are in line with what is seen in the ZMS inventory. Among others we find significant differences for treatment and non-treatment students on the learnability scale and on the social interaction scale. Learnability here means to learn how to use and play the game (Petri, Gresse von Wangenheim, & Borgatto, 2018) and so has an intersection with the simulation variables from the ZMS inventory that ask for comprehension of the game. The structured reflection assignment likely leads to an improved understanding of the sg. In the treatment of our qualitative results we will go deeper into this explanation. The significant difference on the social interaction scale confirms the results from the ZMS inventory regarding teamwork. Both inventories suggest that the structured reflection assignments lead to an improved social experience as a group.

We conclude that the intervention did not influence the perceived quality of the roundly in-between-debriefing. But it has small to medium effects in three areas: treatment students are more engaged, they rate themselves to have a better understanding of the sg and an improved experience as a team. The three described differences may have an impact on the better results and higher assessment for learning that were presented in RQ1.

RQ3: What levels of learning/reflection do we see in the notes on the flips?

In the following we present the results of our qualitative analysis. From gathered answers on the reflection questions we can deduct that students discussed a broad variation of topics. Potential learnings are diverse and evolve during the game play over different game rounds. The teams made clear points on what they thought went well, what they considered not going so well and how they thought they could improve this.

On a regular basis it is visible that goals were achieved and new goals or topics were set every round or previous goals were further specified. These are both indicators for learning, because it is often visible that a preset goal is achieved in the next round or otherwise on the final third flip over. A further specification of a goal can be seen as deepening of insight (Ries, Schaap, van Loon, Kral, & Meijer, 2022). Sometimes it is not visible a goal has been achieved because no references are made toward it, however we do find participants set new goals/topics that could not have been set if they hadn’t solved the other issues. The learning aim of the simulation games used in this study is to manage as a team a large variety of indicators via sharing of information, planning and interpretation of results. Therefore, it makes sense the teams continuously add new topics especially when knowing the complexity of the game increases with more information and indicators as the game rounds proceed.

With RQ3 we address the question on what levels the teams reflected during the game. Looking at the four seminars using the sg GM we see a majority of 67% of first order learning (Table 7). This can be logically explained by the goals of the game. Participants should apply their content and procedural knowledge to the game to be successful. Second order learning accounted for 24 percent. This makes sense because regular reflection on the group process can aid quality in decision making and learning from the multiple perspectives in the cooperation. In some instances also third order reflection took place evoked by the reflections on their personal roles in the group and what they needed to contribute to the organization.

Table 7.

Levels of Learning Per Simulation Game.

For the change management game we see a majority of 43 percent in first order learning (Table 7). This can be explained by the goal of this game, which had less emphasis on applying pre-determined learned norms and rules because the players actually had to find out how to influence the game in such a way they developed an effective approach. Therefore, the process component logically had a more prominent role in this game and accounted for 36% of the cases. While third order learning accounted for another 21%, which makes sense because there was a stronger emphasis on the process and the participants also needed to reflect more on their role in the process and their added value.

A remark has to be made on the sample size which was considerably smaller for STC as opposed to the general management game (eight teams in GM, two teams in STC). Though the sessions were representative to other sessions as confirmed by the experienced facilitators with this game.

What does this mean in relation to the goal of the game? In GM the learning goal for the students was to run a profitable sustainable company. In STC students' learning goal was to move organizational members through the phases of organizational change. The type of learning goal combined with the characteristics of the sg’s and the facilitation interventions have implications for the type of learning taking place but how these actually relate and what the exact implications are is unknown. We state this cautious because learning as intended doesn’t necessarily deliver learning as happened (Leigh, 2003; Wijse-van Heeswijk et al., in press).

For both games we count a majority of first order statements on the reflection notes (e.g., “investment in new machines” or “not reached our sales target”). It shows us that teams are involved in understanding the content of the simulations and finding good decisions. This is in line with the ideas of the games: they are designed and used to apply knowledge and experience the interdependencies of systems. Seeing also a significant number of second and even third order statements emphasizes that students learning with simulation games are active participants in simulated social organizations. They are not only confronted with content but must make organizational decisions. In the third GM seminar one team reflected on the exchange of information within the team. This second order statement can be seen as an example for organizational decisions teams had to made. It can be assumed that decisions on different levels (first, second, third order) interact with each other. The organization of the exchange of information within a team may have great impact on facts that are available for decision making. Having not enough facts (first order) for decision making could lead to the idea to change the organization within the team (second order).

RQ4: What learning process do we see in the teams from round to round?

The reflection-notes on the flips document the group discussion in a chronological manner. So, the notes provide insight into a process of discussion and learning the treatment groups went through. As an example, two out of ten group reflections are elaborated in detail here. For these two examples all data can be found in the appendix (more data are available with the researchers). Next to the examples a summary containing data from all groups is given.

Example 1

GM1 Team A (see appendix):

For this group most of the topics occur only once (24 times, scenario 1). It remains unclear why the groups lost track of the topic. Possibly students learned immediately how to deal with the topic or found the topic irrelevant in later rounds. For scenario 1 it is unclear. We count three issues that were discussed as problematic in reflection one and two (“Calculation of cost of goods sold for copy budget” (1) “Staff utilization too high” (2), “Sales volume estimated too high” (3)) but were discussed as “went well” in reflection 3, which is labeled as scenario 3. In reflection one and two the topic of appropriate staff utilization was discussed as problematic (scenario 4). Only one topic was discussed as satisfying in the beginning (“good share price”) but as problematic in later rounds, which therefore is labeled as scenario 5.

Example 2

STC Team blue (see appendix)

For this team we see seven examples of scenario 1. Scenarios 2 to 4 are found once each and no example of scenario 5 is seen. After round one the team stated that `a plan` was needed. Indeed, it is essential for the simulation to bring 42 action cards into a structured and continuing order. After identifying their lack of a plan, the team obviously found a functional pattern (a plan) because it was mentioned in all later reflections as positive. After round one the team identified a major problem and was able to work on it in the next rounds.

These two examples lead to an inductive hypothesis explaining why we see a better game performance for treatment groups and higher values for self-reported learning: The structured reflection assignment helps students to identify and formulate relevant problems to work on. Formulating issues may reduce complexity and help teams to focus on relevant problems to work on.This hypothesis is confirmed by summarizing the qualitative data. As already seen in the described examples the first scenario is found most for all ten groups (Table 5). Due to the open reflection questions, it is logical to see topics that occur only once or are only relevant for one round. Learners proceed in learning and things that are learned (or do not matter anymore) can disappear from the reflection list. In 20 instances we see teams identifying an issue as problematic but finding ways to deal with it in a positive way in later rounds (scenario three). Identifying an issue as positive in more than one round (scenario two) is found 11 times. Altogether 15 times we see teams naming topics as problematic in more than one reflection round, indicating that teams were challenged by the topic and did not find satisfying solutions for them yet (scenario four). In only three cases we see issues `falling` from positive in earlier rounds to problematic in later rounds.

Looking only at the four scenarios (two to five) that afford a topic to be mentioned at least in two reflections we can state that scenario three appears most frequent (Table 8 following). It confirms the inductive thesis that the consecutive reflection assignment helps groups to identify relevant topics and to work on them successfully.

Table 8.

Summary Learning Process.

Conclusion and Discussions

With a quasi-experimental research design the role of pre structured consecutive reflection questions for learning in sg was researched. In five sg-based seminars treatment groups participated in three reflection interventions (start, middle, end) with three pre structured open questions while non-treatment groups did not. It was found that treatment groups performed better in the games and assessed their self-reported learning higher when asked for concrete learning goals (RQ1). Differences between treatment and non-treatment students are also seen in the assessment of the gameplay process (RQ2). Effect sizes for sg-results and learning (RQ1) and engagement, teamwork and sg-comprehension (RQ2) were small and medium. With regard to the scope of the intervention this is still a remarkable result. In block seminars lasting two to three days three reflection sessions of fifteen minutes each made a difference. Treatment students see themselves to be more engaged in the game and have higher values for teamwork (ZMS inventory) and social interaction (MEEGA+). Also, the learnability (MEEGA+) and the comprehension (ZMS inventory) of the game was rated higher by treatment students, indicating that they had a better understanding of the sgs. Analyzing the reflection notes of treatment groups with qualitative methods we mainly find examples for first order learning and a tendency of more second and third order learning in the change management simulation. It can be assumed that the simulations address different levels of learning (RQ3). Looking at the reflection notes in a chronological order we find reflection-groups to identify relevant problems and to manage them successfully (RQ4).

These results provide a congruent picture. The reflection intervention required group discussion and the results emphasize that treatment students rated their group experience and social interaction higher than non-treatment students. Treatment students rated themselves to have a slightly better understanding of the simulation games which is in line with better self-reported learning and even better success values by the end of the simulation games. Structured group discussion with open reflection questions (What went well?, What went not so well? and What do we need to improve?) may have led to a better understanding of the relevant problems for the simulation which leads to better results. In our qualitative analysis we see further indications explaining why treatment teams have better results and might have profited more from the seminars. The repeated reflection assignment helped teams to identify relevant problems and in many cases the teams were able to work on them properly (RQ4). For GM we reported roundly results (for STC roundly results are not available) and it is obvious that only in the last rounds treatment and non-treatment teams differentiated. This matches with our qualitative results. The reflection assignment serves as a circular learning mechanism. It takes repetition identifying issues or goals, working on them, reflecting them again and then to restart the circle. This is in line and confirms the theory on experiential learning. Experience and reflection complement each other in a circular relationship (Hilzensauer, 2008; Kolb & Kolb, 2009). Interrupting the gameplay with reflection assignments gives students the chance to look back to decision making and to identify relevant issues as shown in Graphic 1. By this process treatment teams gained focus in a complex and confusing environment. The relevant issues found are not only reflected but also anticipated for the next round of decision making. This leads to clearer ideas for action in the next rounds and provides players with agency. We think it is mainly this permanent circle of reflection and anticipation that helped treatment teams to understand the games better and therefore to achieve better results.

Graphic 1.

Process of decision making and reflection.

The overall conclusion of this study is adding structured reflection questions for (in-between-) debriefings during sg sessions adds value to learning.

Limits and further recommendations

Even though the results are congruent and plausible they should be confirmed with further research and more participants/groups. Students participating in this study are all business students from one University in Stuttgart, Germany. To confirm and generalize these results more research is needed applying different contexts, cultures and professions.

We were able to use at least two different simulation games in this study. Further research should evaluate if the presented results are confirmed when applying the reflection assignment to other simulation games.

The discussion notes that were used for qualitative analysis represent only the results of the discussion and give only a narrow insight into the discussion. To understand this even better a deeper insight is needed and further qualitative data of group discussions and gameplay (voice, video) could be collected and analyzed. Especially the reflection notes of one group were almost the same from round to round and researchers were not sure how seriously the group took the discussion.

Supplemental Material

Supplemental Material - The Role of Reflection in Learning with Simulation Games – A Multi-Method Quasi Experimental Research

Supplemental Material for The Role of Reflection in Learning with Simulation Games – A Multi-Method Quasi Experimental Research by Tobias Alf, Marieke de Wijse, and Friedrich Trautwein in Simulation & Gaming.

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iDs

Tobias Alf

Marieke de Wijse

Supplemental Material

Supplemental material for this article is available online.

Notes

Author Biographies

Tobias Alf works as an academic assistant at the Centre for Management Simulation at the Baden-Wuerttemberg Cooperative State University Stuttgart and is a PhD candidate at the Otto von Guericke University in Magdeburg. He is involved in teaching with Simulation Games and conducts educational research on simulation game-based teaching and learning.

Marieke de Wijse has been working as game simulation developer and researcher on simulation games since 2004. Now she is an external PhD candidate at Radboud University, Nijmegen school of Management, section intervention methodology. She researches the effects of facilitator interventions on different types of learning.

Friedrich Trautwein is the scientific director of the Centre for Management Simulation at the Baden Wuerttemberg Cooperative State University Stuttgart. He is also chairman of the jury of the German Simulation Game Award as well as co-responsible for the Simulation Game Research Forum and the European Simulation And Gaming Forum. In 2009, he was awarded the State Teaching Price of Baden-Württemberg for his outstanding and innovative didactic concepts in the field of simulation and gaming.

References

Alklind Taylor

A.-S.

(2014). Facilitation matters: A framework for instructor-led serious gaming. Skövde: University of Skövde.

Argyris

(1977). Double Loop Learning in Organizations. Retrieved from https://hbsp.harvard.edu/product/77502-PDF-ENG

Argyris

Schon

D. A.

(1974). Theory In Practice: Increasing Professional Effectiveness. San Fransisco: Jossey-Bass.

Baker

A. C.

Jensen

P. J.

Kolb

D. A.

(1997). In Conversation: Transforming Experience into Learning. Simulation & Gaming, 28(1), 6–12. https://doi.org/10.1177/1046878197281002

Bartunek

J. M.

(2014). From the Editors: Honoring the Legacy of Chris Argyris by Devoting Attention to How Managers Learn. Academy of Management Learning & Education, 13(1), 1–4. https://doi.org/10.5465/amle.2013.0369

Bedwell

W. L.

Pavlas

Heyne

Lazzara

E. H.

Salas

(2012). Toward a Taxonomy Linking Game Attributes to Learning. Simulation & Gaming, 43(6), 729–760. https://doi.org/10.1177/1046878112439444

Bortz

Schuster

(2010). Statistik für Human- und Sozialwissenschaftler. Berlin, Heidelberg: Springer Berlin Heidelberg. https://doi.org/10.1007/978-3-642-12770-0

Brazhkin

Zimmerman

(2019). Students' Perceptions of Learning in an Online Multiround Business Simulation Game: What Can We Learn from Them? Decision Sciences Journal of Innovative Education, 17(4), 363–386.

Cloutier

Langley

(2020). What Makes a Process Theoretical Contribution? Organization Theory, 1(1), 263178772090247. https://doi.org/10.1177/2631787720902473

10.

Collier

(2011). Understanding Process Tracing. PS: Political Science & Politics, 44(04), 823–830. https://doi.org/10.1017/S1049096511001429

11.

Crossan

(2003). Chris Argyris and Donald Schön's Organizational Learning : There is no silver bullet. Academy of Management Perspectives, 17(2), 38–39. https://doi.org/10.5465/ame.2003.10025187

12.

Deen

(2015). G.A.M.E. Games Autonomy Motivation & Education. Eindhoven: Eindhoven University of Technology.

13.

Duke

R. D.

(2014). Gaming: The future's language. Bielefeld: W. Bertelsmann Verlag.

14.

Eid

Gollwitzer

Schmitt

(2011). Statisitk und Forschungsmethoden. Basel: Belz Verlag.

15.

Friedman

V. J.

Torbert

W. R.

Nielsen

Silverman

Bradbury

(2014). Chris Argyris (1923–2013): In memoriam. Action Research, 12(1), 110–113. https://doi.org/10.1177/1476750313519542

16.

Geithner

Menzel

(2016). Effectiveness of Learning Through Experience and Reflection in a Project Management Simulation. Simulation & Gaming, 47(2), 228–256. Retrieved from http://nbn-resolving.org/urn:nbn:de:bsz:14-qucosa2-353942

17.

Geurts

J. L.

Duke

R. D.

Vermeulen

P. A.

(2007). Policy Gaming for Strategy and Change. Long Range Planning, 40(6), 535–558. https://doi.org/10.1016/j.lrp.2007.07.004

18.

Greenwood

(1988). The role of reflecion in single and double loop learning. Journal of Advanced Nursing, (27), 1048–1053.

19.

Hilzensauer

(2008). Theoretische Zugänge und Methoden zur Reflexion des Lernens. Ein Diskussionsbeitrag. Bildungsforschung, 5(2). Retrieved from http://www.bildungsforschung.org/Archiv/2008-02/lernvermoegen/

20.

Hühn

Rausch

(2022). Collaboration and emotions during simulation-based learning in general management courses. Studies in Educational Evaluation, 73, 101130. https://doi.org/10.1016/j.stueduc.2022.101130

21.

Husebø

S. E.

Dieckmann

Rystedt

Søreide

Friberg

(2013). The relationship between facilitators’ questions and the level of reflection in postsimulation debriefing. Simulation in Healthcare : Journal of the Society for Simulation in Healthcare, 8(3), 135–142. https://doi.org/10.1097/SIH.0b013e31827cbb5c

22.

Klabbers

J. H. G.

(2009). The magic circle: Principles of gaming & simulation (3rd and rev. ed.). Rotterdam: Sense Publishers.

23.

Kolb

A. Y.

Kolb

D. A.

(2009). The Learning Way. Simulation & Gaming, 40(3), 297–327. https://doi.org/10.1177/1046878108325713

24.

Langley

(1999). Strategies for Theorizing from Process Data. Academy of Management Review, 24(4), 691. https://doi.org/10.2307/259349

25.

Langley

Tsoukas

(2016). The SAGE Handbook of Process Organization Studies. 1 Oliver's Yard, 55 City Road London EC1Y 1SP: SAGE Publications Ltd. https://doi.org/10.4135/9781473957954

26.

Lee

J. Y.

Donkers

Jarodzka

Sellenraad

van Merriënboer

J. J.

(2020). Different effects of pausing on cognitive load in a medical simulation game. Computers in Human Behavior, 110, 106385. https://doi.org/10.1016/j.chb.2020.106385

27.

Leigh

E. E.

(2003). A practitioner researcher perspective on facilitating an open, infinite, chaordic simulation : learning to engage in theory while putting myself into practice (Thesis). University of Technology, Sydney. Retrieved from http://hdl.handle.net/10453/20068

28.

Muno

(2016). Fallstudien und Process Tracing in der Vergleichenden Politikwissenschaft. In Lauth

H.-J.

Kneuer

Pickel

(Eds.), Handbuch Vergleichende Politikwissenschaft (pp. 79–90). Wiesbaden: Springer Fachmedien Wiesbaden. https://doi.org/10.1007/978-3-658-02338-6_6

29.

Nakamura

(2022). The Effects of Structured Instruction on Team Performance. In Dhar

Dubey

Dumblekar

Meijer

Lukosch

(Eds.), Gaming, Simulation and Innovations: Challenges and Opportunities (pp. 15–27). Cham: Springer International Publishing.

30.

Pasin

Giroux

(2011). The Impact of a Simulation Game on Operations Management Education. Computers & Education, 57(1), 1240–1254.

31.

Petri

Gresse von Wangenheim

Borgatto

A. F.

(2017). MEEGA+, Systematic Model to Evaluate Educational Games. In Lee

(Ed.), Encyclopedia of Computer Graphics and Games (pp. 1–7). Cham: Springer International Publishing. https://doi.org/10.1007/978-3-319-08234-9_214-1

32.

Petri

Gresse von Wangenheim

Borgatto

A. F.

(2018). MEEGA+: A Method for the Evaluation of Educational Games for Computing Education. Brazil: Universidade Federal De Santa Catarina.

33.

Pospeschill

(2022). Testtheorie, Testkonstruktion,Testevaluation (2nd ed.). München: Ernst Reinhardt Verlag.

34.

Reynolds

(2014). Triple-loop learning and conversing with reality. Kybernetes, 43(9/10), 1381–1391. https://doi.org/10.1108/K-07-2014-0158

35.

Ries

K. E. de

Schaap

van Loon

A.-M. M. J. A. P.

Kral

M. M. H.

Meijer

P. C.

(2022). A literature review of open-ended concept maps as a research instrument to study knowledge and learning. Quality & Quantity, 56(1), 73–107.

36.

Riva solutions GmbH . SysTeamsChange®: Das führende Planspiel zum Veränderungsmanagement. Retrieved from https://www.systeams-change.com/

37.

Schwägele

Zürn

Lukosch

H. K.

Freese

(2021). Design of an Impulse-Debriefing-Spiral for Simulation Game Facilitation. Simulation & Gaming, 52(3), 364–365. https://doi.org/10.1177/10468781211006752

38.

Stoppelenburg

Caluwe

L. de

Geurts

(2012). Gaming: Organisatieverandering met spelsimulaties (geh. herz. editie). Deventer: Kluwer.

39.

Taylor

A.-S. A.

Backlund

Niklasson

(2012). The Coaching Cycle. Simulation & Gaming, 43(5), 648–672.

40.

TOPSIM GmbH . TOPSIM - General Management. Retrieved from https://www.topsim.com/wp-content/uploads/2021/03/SD_TOPSIM_GeneralManagement.pdf

41.

Tosey

Visser

Saunders

M. N. K.

(2012). The origins and conceptualizations of ‘triple-loop’ learning: A critical review. Management Learning, 43(3), 291–307. https://doi.org/10.1177/1350507611426239

42.

Trautwein

Alf

(2022). Theoriebasierte Entwicklung eines Inventars zur Evaluation von Planspielveranstaltungen. In Alf

Hahn

Zürn

Trautwein

(Eds.), Planspiele - Erkenntnisse aus Praxis und Forschung: Rückblick auf den Deutschen Planspielpreis und das Europäische Planspielforum 2021 (pp. 63–87). [S.l.]: Books on Demand.

43.

Visser

(2007). Deutero-Learning in Organizations: A Review and a Reformulation. Academy of Management Review, 32(2), 659–667. https://doi.org/10.5465/AMR.2007.24351883

44.

Visser

Chiva

Tosey

(2018). Guest editorial. The Learning Organization, 25(4), 218–223. https://doi.org/10.1108/tlo-02-2018-0021

45.

Watt

Smith

(2021). Research-Based Game Design for Serious Games. Simulation & Gaming, 52(5), 601–613. https://doi.org/10.1177/10468781211006758

46.

Wijse-van Heeswijk

M. d.

Rouwette

van Laere

(in press). A case study report on facilitation interventions to increase learning effectiveness in game simulations. In Angelini (Ed.), Simulation for Participatory Education. Virtual Exchange and Worldwide Collaboration. Springer Fachmedien Wiesbaden.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.14 MB