Gamifying Online Training in Management Education to Support Emotional Engagement and Problem-solving Skills

Abstract

Online training to improve problem-solving skills has become increasingly important in management learning. In online environments, learners take a more active role which can lead to stressful situations and decreased motivation. Gamification can be applied to support learner motivation and emotionally boost engagement by using game-like elements in a non-game context. However, using gamification does not necessarily result in supporting positive learning outcomes. Our analysis sheds light on these aspects and evaluates the effects of points and badges on engagement and problem-solving outcomes. We used an experimental approach with a fully randomized pre-test/post-test design of a gamified online management training program with 68 participants. The results demonstrate that points and badges do not directly improve problem-solving skills but are mediated by emotional engagement to positively influence problem-solving skills. Additionally, satisfaction with the gamification learning process positively relates to emotional engagement. Thus, when creating online training programs, it is essential to consider how to engage students and to think about the design of the learning environment. By identifying the limitations of gamification elements, the study’s results can provide educators with information about the design implications of online training programs for management learning.

Keywords

gamification learning emotional engagement problem-solving skills satisfaction management training goal orientation

Introduction

Online learning has become increasingly important over the past two decades which has created new challenges for learners (Bughin et al., 2018). In online learning, management students take a more active role in the learning process and have more responsibilities (Wan et al., 2012), possibly resulting in stressful situations that can lead to decreased motivation and higher drop-out rates (Nawrot & Doucet, 2014; Omar et al., 2009). Moreover, complex online learning environments pose challenges related to motivation and engagement; learners often disengage when confronted with more open-ended tasks (Aparicio et al., 2019; Hanus & Fox, 2015).

Training for problem-solving skills can be particularly challenging in management education. Problem-solving outcomes are especially important for management students because they require that students learn how to resolve unstructured, complex problems (Bigelow, 2004; Janson et al., 2020; Peterson, 2004). Problem-solving skills are defined as “situated, deliberate, learner-directed, activity-oriented efforts to seek divergent solutions to authentic problems through multiple interactions amongst problem solvers, tools, and other resources” (Kim & Hannafin, 2011, p. 405) and they are considered to be essential in an ever-changing business world.

Because it can be difficult to teach problem-solving skills to management students, especially online, learners require some assistance. Gamification is a concept that can help students improve their problem-solving skills and enhance the outcomes of their tasks. Gamification entails deploying elements familiar to online gaming contexts within a non-entertainment context (Deterding et al., 2011; Hamari et al., 2014) to assist learners in staying engaged with and focused on their activities, such as problem-solving tasks. Using gamification elements can have positive effects on learning outcomes—especially concepts that support lower level learning outcomes such as multiple choice quizzes (Dias, 2017). However, the impact of gamification elements on problem-solving outcomes has been questioned because their effects can be positive or negative, creating a divide between researchers and calling their benefit into question (Kalogiannakis et al., 2021). It has been proposed that gamification can emotionally support learner engagement, ending in better outcomes (Hammedi et al., 2021). Although gamification has been intensively discussed in the education literature (Schöbel et al., 2021), analysis of the theoretical and empirical issues related to the overall gamification context remains incomplete (Koivisto & Hamari, 2019). Researchers ask for more experimental studies that analyze how engagement relates to other gamification and learning related variables to better explain its effect and meaning (Hamari et al., 2016; Schöbel et al., 2020). Therefore, the study discussed in this paper focused on the following research question:

RQ: How does gamification relate to emotional engagement and problem solving in management education?

To answer this research question, we conducted a randomized pre-test/post-test experiment in a management course and, for this purpose, we developed a gamified online training program to teach problem-solving skills. Our results demonstrate that although the direct effect of gamification on problem-solving skills was not significant, gamification operates through emotional engagement to influence problem-solving skills. Thus, our study contributes to management learning literature by establishing emotional engagement as an important mediator in supporting the effects of gamification on problem-solving. By highlighting emotional engagement as the mechanism through which gamification influences problem-solving skills we are able to explain how gamified online training, points, and badges can be used to support emotional engagement and ultimately problem-solving skills. In addition to emotional engagement, we establish learning process satisfaction as an important construct that can better support problem-solving skills through its effect on emotional engagement when using gamification in online learning. From a practical perspective, we present how badges and points that are integrated in online trainings should be designed to effectively support learning by reinforcing emotional engagement in and satisfaction with the learning process.

Theoretical Foundations

In the following, we start by defining and explaining gamification compared to other relevant terms such as simulations. Next, we present the elements that can be used to gamify online trainings. Finally, we clarify which elements we used for this study and why.

Gamification in Online Trainings

In management education, we have three different approaches to make learning more engaging with the concepts of games: simulations, serious games, and gamification. A simulation is a synthetic or artificial environment that supports individuals in experiencing reality. It can be used during business development to mimic a business situation (Salas et al., 2009). In a simulation, we can use specific game elements to engage learners in interacting in a business situation. Alternatively, serious games are described as a mental contest, often played with a computer in accordance by specific rules. Serious games use entertainment to further government and corporate training, as well as education, health, public policy, and strategic communication objectives (Sicart, 2014; Zyda, 2005). In serious games, we have a virtual world that is close to those that we have in online games, and elements of games are placed within the virtual world that learners are operating in. However, we can use elements of a game also without a virtual simulation or a virtual world. If we use only elements of a game without embedding them in a virtual world or simulation, we use the concept of gamification. In other words, gamification is the use of game-like elements in non-gaming contexts (Deterding et al., 2011).

Gamification has been used in several different contexts, such as finance, health, education, sustainability, and productivity (Deterding et al., 2011; Fernandes et al., 2012). In the realm of learning, gamification has the following four general purposes: supporting a process, enabling a behavior, supporting in processing information, and creating a business value (Treiblmaier et al., 2018). Gamification supports the learning process by making something ordinary more enjoyable. In other words, gamification can create the joy of application use and can accomplish mastery and autonomy. Another purpose of gamification is to support the processing of information (Treiblmaier et al., 2018). Information processing involves solving problems, learning, or changing learning-related behaviors and attitudes (Treiblmaier et al., 2018). From a business perspective, gamification can assist in making products or services more engaging, thus resulting in greater values (Treiblmaier et al., 2018).

Compared to other concepts, gamification has several advantages. In online learning, learners have to organize their learning process, which makes it necessary for them to understand the reason behind what they are doing (Leemkuil & de Jong, 2012). Training management students to solve daily business problems requires keeping them motivated and engaged in the learning process (Swain et al., 2020). Gamification can make learning more entertaining, helping students to control their actions in a digital learning environment and bring pieces of a puzzle together (Liu et al., 2011). At the same time, from a teacher or managers perspective, gamification is an effective concept that is easy to implement, and inexpensive (Sailer & Homner, 2020).

Elements of Gamification

The elements are an essential part of gamification and can be described as their building blocks (Blohm & Leimeister, 2013), which the designer selects, adapts, and implements in a learning application (Schöbel et al., 2020). Existing gamification approaches are often criticized because of poor planning of the gamified environment, such as the ad hoc use of gamification elements (Kalogiannakis et al., 2021). Moreover, gamification designers have often been criticized for using certain pre-existing patterns of design elements with presumed motivational effects, regardless of the contexts of use (Schöbel et al., 2019a, 2019b). It is commonly agreed that contextual factors are critical for designing gamified systems that support actual learners’ needs (Dichev et al., 2020). Therefore, we describe in the following which gamification elements we selected for this study and how to adapt them to the context of digital learning.

Interaction is typically necessary in open-ended and problem-based learning scenarios to keep learners motivated continuing with their learning processes (Hrbackova & Suchankova, 2016). When considering gamification as an intervention, interaction is possible by considering aspects of feedback. Feedback can be briefly described as a process through which learners make sense of information from different sources (such as peers or a computer-based system) to enhance their work or learning strategies (Carless & Boud, 2018). Feedback helps learners to regulate their learning process because it assists them in combining knowledge to solve complex problems as well as interpreting information (Leemkuil & de Jong, 2012; Wood et al., 1976). Achievement, on the other hand, provides guidance to learners, which is important because they act on their own in online learning environments (Eom & Ashill, 2016; Kuo et al., 2012). We created Table 1 to present the various elements of gamification and explain how each influences feedback and achievement. A “+” indicates that an element can act as a feedback element, whereas a “0” indicates that changes are necessary to support an element’s function of giving feedback. A “−” indicates that giving feedback is not possible with that element.

Table 1.

Gamification Elements and Their Related Influence on Feedback and Achievements.

Element	Description of element	Feedback					Achievement
Element	Description of element	Response	Way of feedback	Actual data	Time	Challenge	Clear	Desirable	Goal
Points	A numerical unit, which can be obtained for completing an activity and/or action.	+	+	+	+	0	+	+	−
Badges	A visual icon that signifies an achievement a user makes while completing an activity.	+	+	+	+	+	+	+	+
Leaderboard	Enables a user to compare his own performance against the performance of other users. The relative position is often presented as a ranking.	−	0	+	−	+	−	0	−
Levels	Shows a user’s progress in working on system activities or actions. It displays his experience through different level positions and builds up.	−	0	+	−	+	0	0	−
Narratives	Narratives are used to tell a story and to generally couch every action or activity.	−	0	−	−	−	−	−	−
Progress bar	Used to indicate the user’s progress in completing activities.	−	0	+	−	−	0	0	−
Reminder	Used to visualize the user’s past behavior by presenting him with a history of his actions.	−	0	+	−	−	−	−	−
User Avatar	Used as a visual representation of the user to personalize their user profile.	−	−	−	−	−	−	−	−
Goals	Achievable steps that users can accomplish while concentrating on actions and/or activities.	+	0	+	0	+	+	+	+
Time pressure	Time pressure is created on activities and actions by using a counter or an hourglass.	−	0	−	−	+	−	−	−
Virtual goods	Assets with a perceived value that can be purchased or traded: e.g., coins.	+	+	+	+	+	+	+	−

= feedback possible; 0 = neutral, feedback possible in parts, − = not possible.

Achievement is described with three characteristics: goal, clear, and desirable (Thiebes et al., 2014). Looking at the elements in Table 1, we can see that badges and goals typically satisfy the achievement characteristic of goal. Four elements—points, badges, goals, and virtual goods—satisfy the achievement characteristics of clear and desirable. Feedback has the following five characteristics that support social constructivism and meta-cognitivism in learning (Thurlings et al., 2013): response, way of feedback, actual data, time, and challenge. Knowledge of response refers to informing learners when they have done well. It can be provided by using points, goals, badges, or goods. For example, a user can be rewarded with a point for giving the correct answer (De-Marcos et al., 2014).

The way of giving feedback to learners is usually either positive or neutral (Thurlings et al., 2013). Most elements provide neutral feedback by showing users their progress, such as a progress bar (Silpasuwanchai et al., 2016). Other elements, such as points or badges, can be positively experienced by rewarding users for being successful when completing a learning activity (Davis & Singh, 2015). Actual data refers to the element’s ability to present the actual data related to a learner, which most elements fulfill. Time refers to the timing of giving feedback to a learner, which can be either formative or summative (feedback during a learning process versus consolidated feedback at the end of a learning process). Points, badges, and goods can be offered to learners in both ways by giving a point to learners for each time they have completed a test question or by giving a badge to learners for giving the correct answer to a series of questions (Davis & Singh, 2015; De-Marcos et al., 2014). Finally, feedback is connected to challenges (Thurlings et al., 2013) and an element’s ability to support learners in surpassing their prior achievements. This holds for almost all elements.

For this study, we consider points and badges for the following reasons. Having an overall point score signals how well learners have performed in completing a given learning process. Such signaling is important in online learning because learners have a more active role and need to be supported in developing competencies on their own, which can be partly done by the use of gamification elements (Eom & Ashill, 2016). In line with this, badges support learners in completing achievements, and they provide them with specific feedback.

Hypotheses Development

In this section, we discuss our research model (shown in Figure 1). This model shows our theoretical constructs related to gamification (satisfaction with the learning process, emotional engagement, problem-solving outcomes) and our moderators (intrinsic goal orientation and extrinsic goal orientation).

Figure 1.

Theoretical model.

Gamification to Support Problem-solving Outcomes and Emotional Engagement

Problem-solving skills are important for the careers of business students; however, to date, students often seem to deal poorly with unstructured problems (Bigelow, 2004). Learning to solve complex problems is especially needed in management education (Grossman et al., 2013) to prepare up-and-coming managers for complex work environments (Bowman, 2019). In online environments, learners have to organize their learning process and they are required to manage complex problems on their own (Peterson, 2004). Gamification can be used to support the learning of problem-solving skills by signaling continuous achievements and progress and activating self-regulation (Hanus & Fox, 2015; Jang, 2008; Newmann, 1992). Gamification can make it easier to learn about complex problems by reducing the difficulty of the learning materials and making learning more fun (Buil et al., 2020; Leemkuil & de Jong, 2012). For example, points can be used to simultaneously regulate and reward learning progress. A regulated learning process can result in better problem-solving abilities (Eom & Ashill, 2016; Kim & Hannafin, 2011) by enabling well-coordinated learning processes. Therefore, gamification can help learners better focus on their learning material by providing feedback that supports them in understanding their progress, which signals achievement (Davis & Singh, 2015; Leemkuil & de Jong, 2012). Thus (and as shown in Figure 1), we hypothesize:

H1: Gamification in online learning has a positive effect on problem-solving outcomes.

While learning is not always enjoyable, it can be (Shernoff, 2013). Creating a gamification concept for digital learning environments can increase emotional engagement (Özhan & Kocadere, 2020) which refers to one’s feeling of enjoyment with a learning activity based on it being considered fun and entertaining (Hamari et al., 2014). Thus, emotion plays a valuable role in experiential learning (Taylor & Statler, 2014). Gamification concepts facilitate a feeling of having fun to assist students in achieving a higher level of concentration, which keeps them emotionally engaged (Khan et al., 2017; Newmann, 1992). By emotionally involving learners in a learning process, such as by rewarding them, we can better support them in engaging in an activity for their own sake (Guzzo, 1979). Based on this, we hypothesize (as shown in Figure 1):

H2: Gamification in online learning has a positive effect on emotional engagement.

Learning Process Satisfaction with Gamification

Learning involves statements of opinions and beliefs or an assessment of worth (Pintrich & De Groot, 1990) known as affective learning outcomes (Gupta & Bostrom, 2013). An affective outcome describes the extent to which learners believe the learning and teaching system they are using meets their expectations (Eom, 2014). Satisfaction is typically outcome oriented, for example, satisfaction could be the result of the completion of learning goals, which can be either positive or negative. Learning process satisfaction describes the favorable and unfavorable response of leaners with regards to different characteristics of the training. A satisfying learning process refers to a learning environment where learners perceive the training as being efficient, coordinated, clear, and fair. Gamification can lead to satisfaction with the learning process by highlighting and rewarding the learners progress in online training to stimulate positive feelings about the training (Shute et al., 2015). With gamification, instructors can create a feeling of being satisfied by informing learners about their performance throughout the learning process and by making the learning experience more efficient (Ferguson & DeFelice, 2010; Fisher, 2003). Thus (as shown in Figure 1), we hypothesize:

H3: Gamification in online training has a positive effect on satisfaction with the learning process.

Being satisfied with a learning process is the psychological state of being able to achieve success and having positive feelings about that success and the ability to act most appropriately (Hui et al., 2007; Keller, 1987; Özhan & Kocadere, 2020). In turn, this will make learners want to continue their learning process, which can be observed as greater emotional involvement (Ferguson & DeFelice, 2010; Lee et al., 2011; Özhan & Kocadere, 2020). Satisfied learners can better interpret information, judge their progress, and determine the steps they need to take to achieve a specific learning goal (LG) (Eom & Ashill, 2016; Leemkuil & de Jong, 2012). Overall, learners do best when they are satisfied with a learning process; thus, they enjoy what they are doing and experience greater engagement with the process. This engagement can be determined by learning process satisfaction and the feeling of a coordinated and understandable learning environment (Watson & Sutton, 2012). Therefore (and as shown in Figure 1), we hypothesize:

H4: Learning process satisfaction in online training has a positive effect on emotional engagement.

Emotional Engagement to Support Problem-solving Outcomes

If learners are emotionally engaged in an activity, they are more likely to be engrossed in the process (Parent & Lovelace, 2015). Thus, learners who invest emotional energy in their learning can enhance their learning outcomes (Rich et al., 2010). Related work on emotional engagement highlights that learners engaged in the learning environment display a higher level of success and a lower prevalence of dropping out (Carini et al., 2006; Özhan & Kocadere, 2020). A person who is emotionally engaged is better able to focus on a task, and is better able to solve complex problems. By being able to continuously focus on the most important content and keep learning without giving up or being distracted, learners can understand the relationship between the different concepts they are learning (Ding et al., 2017). Being emotionally engaged can be key to accomplishing complex problems and persevering, even though the learning materials might be complex or difficult to understand, because students enjoy what they are doing (Arbaugh, 2000; Eseryel et al., 2014). Moreover, learners who enjoy being in a digital environment feel happy and pay attention to the given task; therefore, they are more proactively engaged in solving problems (Özhan & Kocadere, 2020). Thus (as shown in Figure 1), we hypothesize:

H5: Emotional engagement in online learning has a positive effect on problem-solving learning outcomes.

The Mediating Effect of Emotional Engagement

Generally, emotional engagement enables learners to focus on their learning process (Kuo & Chuang, 2016). We hypothesize that emotional engagement is also an important construct that mediates the effects of gamification on problem-solving learning outcomes. Being emotionally engaged in learning can better explain the relationship between gamification, its elements, and problem-solving outcomes by signaling the achievements that learners can attain and by showing them the progress they have made in an environment in which they play a more active role and are more responsible for their actions (Kim & Hannafin, 2011). We posit that, with a guiding and emotionally supportive design, gamification elements can facilitate learners’ emotional engagement, simultaneously making learning more entertaining and more efficient. Consequently, learners are supported in better regulating their learning process and in better discovering knowledge on their own (Eom & Ashill, 2016). Thus (as shown in our model in Figure 1), we hypothesize:

H6: Emotional engagement mediates the effect of gamification on problem-solving outcomes.

The Moderating Effects of Intrinsic Goal Orientation and Extrinsic Goal Orientation

We expect that intrinsic goal orientation and extrinsic goal orientation moderate the relationship between gamification and emotional engagement. Research supports the idea that the goal orientation of learners can influence the power of the relationship between gamification and emotional engagement (Super et al., 2019). Intrinsic goal orientation reflects the learners’ perceptions of why they are engaging in an activity (Pintrich et al., 1991). Learners with a strong intrinsic goal orientation have a strong willpower to participate in an activity (Kwak et al., 2021). Such strong willpower can strengthen the relationship between gamification and emotional engagement. In other words, learners with a strong intrinsic goal orientation will be more emotionally engaged in gamification training because they have a stronger desire to learn. Extrinsic goal orientation is about participating in a task to be rewarded (Pintrich et al., 1991). Learners with a strong extrinsic goal orientation have a strong need to get rewarded for their activities and thus will be more emotionally engaged in the gamification learning process because of the opportunity to earn badges and points (Kwak et al., 2021; Özhan & Kocadere, 2020). Therefore, in our research model, we aim to shed light on the boundary conditions of learner goal orientation (intrinsic or extrinsic) between gamification and emotional engagement. Consequently (as depicted in Figure 1), we hypothesize:

H7: Extrinsic goal orientation is expected to positively moderate the relationship between gamification and emotional engagement.

H8: Intrinsic goal orientation is expected to positively moderate the relationship between gamification and emotional engagement.

Research Design and Method

Study Context and Participants

We evaluated our model by conducting a fully randomized, pre-test/post-test, between-subjects experiment with management students at a western European university. We developed an online training program concerning the value proposition canvas (VPC) concept (Osterwalder et al., 2014). A VPC is a tool that can help ensure that a product or service is positioned around what the customer values and needs. A VPC considers the value proposition of a company and assists in creating a customer profile. From a company’s perspective, gain creators, pain relievers, and product and service assets are specified. Pains, gains and the customer’s job are specified in the customer profile. The training was embedded in the course “Business and Information Systems Engineering,” typically attended by 80 to 150 freshman undergraduate students. Participation was voluntary but embedded in the mandatory tutorial sessions on campus. To avoid common method variance (CMV; variance that is attributable to the measurement method rather than to the constructs the measures represent), we did not reveal the goal of our study to our participants but instead provided a cover story and other ex-ante remedies according to Podsakoff et al. (2003). A Harman’s single factor test ex-post showed that CMV should not be a problem in our study.

We collected 68 valid datasets, indicating a response rate of 91.89%. Our non-gamified group consisted of 32 students, and our gamified group consisted of 36 students. In terms of majors, 48 students (70.60%) studied business and economics, 15 (22.10%) studied business law, 3 (4.40%) studied pedagogy, 1 student studied industrial engineering, and another one studied an unlisted course. Our sample consisted of 39 female (57.35%) and 29 (42.65%) male students. Their average age was 22.22 years (see Table 2).

Table 2.

Demographic Data: Age and Prior Knowledge.

Participants	n	Mean	Std. deviation	Median
Age—Total	68	22.22	3.189	21
Age—Gamified Group	36	22.55	3.9
Age—Group without Gamification	32	21.84	2
Prior Knowledge—Total	68	6.764	2.9985	7.00

Online Training and Learning Goals

To analyze our model (Figure 1), we developed an online training program. We began by developing LGs for our online training. Because the instructional environment must represent the learner’s context and learning environment (Gibbons et al., 2000), we decided to guide their cognitive processes, such as analyzing, evaluating, and creating (Anderson & Krathwohl, 2016; Jonassen, 2000).

We focused on three different kinds of problem solving (Jonassen, 2000). Troubleshooting problem solving refers to participants’ abilities to solve case problems by, for example, finding a mistake in a described situation or context. Decision-making problem solving requires the participants to select a single option from a set of alternatives by, for example, identifying the weaknesses and strengths of an existing VPC. And finally, case analysis problem solving refers to using alternative solutions for a given problem that goes beyond using learned methods and techniques by, for example, suggesting alternative canvases to make a business model more valuable for a company.

In accordance with Anderson and Krathwohl (2016) and their updated version of Bloom et al.’s (1956) LG taxonomy, we formulated three overarching LGs concerning type of knowledge and problem-solving skills, as follows:

LG1: The participants can apply their knowledge about the VPC to the given case. This involves conceptual knowledge and story problem solving.

LG2: The participants can analyze and evaluate the VPC and the value proposition of a company. This involves procedural knowledge and troubleshooting and decision-making problem solving.

LG3: Based on their analysis, the participants can evaluate a future value proposition and create an informed investment decision. This involves procedural knowledge and decision-making problem solving, as well as case-analysis problem solving.

We then developed the training in three steps. First, we started with a rough concept of our online training. Second, we developed a storyboard. Third, we transferred our storyboard to our online training program using Adobe Captivate as the development tool. In online training, background information should always be available to learners to support them in achieving better learning outcomes (Leemkuil & de Jong, 2012) by, for example, providing additional material in a given learning process. Constructing online training, requires concentration on conveying the message better to get the ideas across (Gibbons, 2003), which is why we decided to split each learning unit into the following three parts: absorb, do, and connect (Horton, 2011). Absorb concerns a passive activity, such as reading a text; do concerns using what was learned during the absorb process to solve a task; and connect concerns by combining the learned aspects with the participants’ own experiences.

Experimental Manipulation

Points were given to participants to reward their progress in the training rather than for giving correct answers on knowledge tests to avoid demotivation (Santhanam et al., 2016). We also gave out the following two different types of badges: trophy badges were given to participants after completing each unit, and regular badges were given only to participants who viewed supplementary material that was not necessary to finish the online training but suggested for successful completion. The participants were informed about the option of collecting additional rewards at the beginning of the training.

In online training environments, it becomes more important to provide feedback to learners about their performance (Leemkuil & de Jong, 2012). We included audible feedback to highlight that participants had received rewards in the form of points or badges (Li et al., 2012). The collected points and badges were visible to the participants from the first moment they were gained. Points were summarized and displayed as an overall score (Hiltbrand & Burke, 2011). To analyze the effects of gamification, we used: a gamified group with points and badges and a non-gamified group without any gamification elements. Participants were randomly assigned to one of the two groups. The structure of our online training and the reward concept is presented in Figures 2 and 3.

Figure 2.

Experimental procedure.

Figure 3.

Screenshots of training.

Our experiment entailed three parts. The first part consisted of a pretest to determine the prior knowledge and demographics of our participants. The second part was the actual training, which consisted of a repetition unit to learn the basics of the VPC, two learning units, and a case study to measure our learning outcomes. In the third part, the participants had to fill out a questionnaire that was used for the analysis of our hypotheses. The training lasted between 90 and 100 minutes.

Study Measures

To measure problem-solving learning outcomes, we constructed tasks that participants had to solve after completing the second learning unit. For the tasks, we developed a case description by presenting different companies and their VPCs to the participants along with additional information such as company news and other relevant information concerning the cases.

We measured all other dependent variables with established scales and where necessarily adapted scales to our research context. We measured intrinsic and extrinsic goal orientation by using the scale developed by Pintrich et al. (1991), with four items for each construct. The emotional engagement was measured with three provided by Hamari et al. (2014). Two items from the original scale of five items were removed because of their loading below 0.7. We used a bipolar scale from Gupta and Bostrom (2013) to measure the learners’ learning process satisfaction concerned with the use of internet technology (IT) (Chin, 1998) instead of relying on typical measures for affective outcomes and reactions of learners (Brown, 2005). We used four out of five items to measure satisfaction with the learning process. Appendix 1 summarizes our measurement instruments.

In addition, we measured control variables that relate to individual differences in learning behavior, which may influence the outcome of our results. We measured technology readiness by using scales from van der Rhee et al. (2007). We measured participants’ prior knowledge with four test questions about the VPC with a maximum of 10 points that could be collected.

Data Analysis

To evaluate the research model of this study, we used the variance-based partial least squares (PLS) approach. PLS analysis is a multivariate statistical method that allows comparison between multiple response variables (also called constructs) and multiple explanatory variables (in our model, problem-solving outcomes). PLS is one of several covariance-based statistical methods that are often referred to as structural equation modeling, and it was created to deal with multiple regression. In other words, PLS is a regression method that allows for the identification of the underlying factors, which are a linear combination of the explanatory variables, and what we also know as latent variables, which best model the response.

We decided to work with PLS because it is more suitable than other covariance-based methods due to mainly three reasons (Hair et al., 2010). First, PLS is better suited for evaluating data sets with smaller sample sizes that are imposed through our experimental setting in the field. Our sample size (n = 68) is sufficient for the PLS approach using power analyses (Hair et al., 2016). With two constructs pointing at our dependent variable, we would need 33 observations to achieve a statistical power of 80% for detecting R² values of at least 0.25, with a 5% probability of error (Hair et al., 2016). Second, we want to provide explorative evidence on how to better support learners in online learning by referring to elements of gamification in management education. Third, model identification issues of covariance-based approaches can arise when using single-item measures, such as our learning outcome scores.

We developed a coding scheme to analyze the problem-solving learning outcomes of our tasks. Because the tasks were based on different LGs concerning the application of knowledge as well as analysis and evaluation to solve a problem, the tasks required an increasing degree of problem-solving skill, and the scoring was adapted likewise, specifically, 20 points for the first task, and 70 for the second. Two raters were involved in analyzing the learning outcomes. The raters were trained on the coding scheme by the first two authors. Our analysis showed that the raters had a high interrater reliability (IRR) (weighted Cohen kappa = 0.706, p < .05; Pearson correlation coefficient; r = .772; n = 68; p < 0.001). Due to the complexity of rating the problem-solving outcomes, the raters resolved any ambiguities on their own by discussing until they both agreed on a single consensus score, which we then used for further analysis¹

Results

Control Variables and Manipulation Check

Our control variable technology readiness had no significant influence on problem-solving outcomes (t-value = 1.304; p > .10). Additionally, we evaluated the participants’ prior knowledge. The average prior knowledge was 6.46 in the gamified group and 7.08 in the non-gamified group, showing that prior knowledge did not significantly differ between the two groups (p > .10).

Because we conducted an experiment, we wanted to check whether the gamification elements we used were recognized. For this check, we asked our participants three questions (marked on a 5-point Likert scale from “1 = Totally Disagree” to “5 = Agree”). We explicitly ask them if they recognized the gamification elements of points, trophy badges, and regular badges. The results of a t-test for independent variables revealed that all three manipulations were recognized by the participants of our gamified group. We also added these questions to our control group. The participants of our non-gamified group did not recognize any of the elements (for all three manipulation checks p < .001; mean points: gamified group: 4.66, non-gamified group: 1.37; mean badges: gamified group: 4.50, non-gamified group: 1.10; mean trophy badges: gamified group: 4.71, non-gamified group: 1.24).

Research Model Evaluation

To evaluate our model and to test our hypotheses, we analyzed the outer model and continued with the inner model (Hair et al., 2010). The inner model is the part of the model that describes the relationships among the latent variables that determine the model (Hair et al., 2014). The outer model is the part of the model that highlights the relationships among the latent variables and their indicators (Hair et al., 2014). A part of this involves an analysis of reliability and validity. We evaluated the reliability and validity of the outer model using quality criteria (Table 3).

Table 3.

Quality Criteria of Measurement Model.

Construct	Indicator	Loading	AVE	Composite reliability	Mean
Learning outcomes*	Problem solving Skill	1.00	-	-	Overall: 6.69 (S.D.: 3.03)
					Treatment: 7.25 (S.D.: 3.03)
					Control: 6.02 (S.D.: 3.05)
Gamification*	Group	1.00	-	-	-
Emotional engagement**	E3	0.939	0.877	0.955	Overall: 2.66 (S.D.:0.948)
	E4	0.926			Treatment: 3.06 (S.D.: 0.800)
	E5	0.945			Control: 2.20 (S.D.: 0.906)
Learning process satisfaction**	Satis1	0.833	0.676	0.893	Overall: 4.36 (S.D.:1.12)
	Satis2	0.858			Treatment: 4.66(S.D.: 0.84)
	Satis3	0.719			Control: 4.03 (S.D.: 1.31)
	Satis5	0.871			Control: 4.03 (S.D.: 1.31)

Constructs were operationalized with one indicator. Therefore, AVE and composite reliability could not be computed.

E1 (0.608), E2 (0.595), and Satis4 (0.683) were removed due to loading below 0.7.

Indicator reliability was analyzed with indicator loadings. We used indicator reliability to analyze the proportion of indicator variance that is explained by our latent variables and that can range from 0 to 1 (Hair et al., 2014). The indicator loadings should be above the minimum value of 0.7 (Hulland, 1999). We measured internal consistency by referring to the means of construct reliability; values should be above 0.70 to have acceptable construct reliability (Bagozzi & Yi, 1988), and ours were.

Furthermore, we measured convergence validity using the average variance extracted (AVE). Generally, convergence validity is a subtype of construct validity and assisted us in evaluating the degree to which two measures that should be related as claimed by theory that are related (Hair et al., 2014). The value should be above the minimum of 0.50 so that at least half the variance of the constructs is explained by the measured indicators (Bagozzi & Yi, 1988). All of our constructs have AVEs above 0.50.

We then used AVE to measure discriminant validity (see Table 4). We used discriminant validity to test the degree to which our measure diverges from another measure. This can happen by measuring the Fornell-Larcker criterion, which states that the square root of the AVE of a construct should be higher than the correlation of the latent construct with other constructs of the measurement and indicates whether a construct shares more variance with its indicators than with other constructs (Fornell & Larcker, 1981). In another step, we assessed the heterotrait-monotrait (HTMT) ratio and the HTMT interference criteria (HTMT_inference). HTMT interference is a measure of similarity between our latent variables (Hair et al., 2014). Discriminant validity was established through consideration of the Fornell-Larker criterion and the conservative HTMT85 measure (indicated through all HTMT measures being under 0.85) (see Table 4). The HTMT_inference values are all significantly below the threshold of 1.

Table 4.

Discriminant Validity*.

Construct	(1)	(2)	(3)	(4)
(1) Problem solving learning outcomes	NA
(2) Gamification	0.197 (0.197)	NA
(3) Emotional engagement	0.384 (0.470)	0.453 (0.470)	0.936
(4) Learning process satisfaction	0.391 (0.433)	0.282 (0.310)	0.608 (0.678)	0.822

The square roots of AVE are in bold. The off-diagonal elements are correlations of latent variables. The computation of the Fornell-Larcker criterion was omitted for the manifest variables (NA). Values in parenthesizes show the HTMT criterion, where 0.85 represents a conservative threshold. Therefore, the values show that the conservative HTMT85 criterion is fully satisfactory and confirming discriminant validity (Henseler et al., 2015).

Lastly, to analyze our outer model, we analyzed cross-loadings (Chin, 1998) that show all indicators load the highest on their constructs, (see loadings in bold in Table 5) (Hair et al., 2014); this can be seen in Table 5. Simply saying this provides insurance to us that our indicators clearly can be assorted to our latent variables and not the other ones.

Table 5.

Cross Loadings.

Indicator	Engagement	Gamification	Problem solving	Learning process satisfaction
E3	0.939	0.372	0.399	0.592
E4	0.928	0.441	0.285	0.494
E5	0.945	0.461	0.383	0.611
Gamification	0.453	1.000	0.197	0.282
Problem solving-skill	0.383	0.197	1.000	0.391
Satis1	0.477	0.159	0.478	0.833
Satis2	0.561	0.231	0.290	0.858
Satis3	0.395	0.309	0.246	0.719
Satis5	0.544	0.232	0.288	0.871

After we evaluated the outer model, we evaluated the inner model to determine which of our hypotheses can be supported and which we need to reject. The results of the structural model can be seen in Figure 3. Hypotheses with a p-value less than or equal to .05 were accepted. Those with a p-value greater than or equal to .05 were rejected. In the remainder of this section we detail how we analyzed our inner model to establish the path coefficients, R², significance levels, effect sizes, and predictive relevance (Hair et al., 2016). Table 6 summarizes our results, stating each hypothesis and whether it was supported.

Table 6.

Summary of Results.

Hypothesis		Path coefficient	t-Value for path	Support for hypothesis
H1	Gamification → (+) Problem solving learning outcome	0.029	-	No
H2	Gamification → (+) Emotional engagement	0.306	2.933	Yes
H3	Gamification → (+) Learning process satisfaction	0.282	2.575	Yes
H4	Learning process satisfaction → (+) Emotional engagement	0.521	5.702	Yes
H5	Emotional engagement → (+) Problem solving learning outcome	0.371	2.890	Yes
H6	Gamification × Intrinsic goal orientation (+) → Emotional engagement	0.024	-	No
H7	Gamification × Extrinsic goal orientation (+) → Emotional engagement	−0.230	-	No
H8	Gamification → (+) Emotional engagement → (+) Problem solving learning outcomes	Mediation analysis		Yes

For Hypotheses 1 to 7, we report the path coefficients and their p-values for the direct effects along with their significance level. The results can be found in Figure 4. Three of our hypothesized relationships are not significant (their p-value is above .05). There is no significant relationship between gamification and problem-solving learning outcomes (ß = .030; p > .05) (H1). Moreover, our PLS analysis also highlights that the moderating effects of intrinsic goal orientation and extrinsic goal orientation on the relationship between gamification and emotional engagement are not significant (ß = .024, p > 0.05; ß = −.230, p > 0.05) (H6 and H7, respectively).

Figure 4.

Results of structural model.

All other hypotheses are supported and are reported in order from largest path coefficient to smallest path coefficient. Our results suggest that emotional engagement positively relates to problem-solving outcomes (ß = .371, p < .01) (H5). Learner process satisfaction positively relates to emotional engagement (ß = .521, p < .01) (H4), and gamification positively relates to emotional engagement (ß = .306, p < .01 ) (H2), and positively to learning process satisfaction (ß = .282, p < .01) (H3).

To analyze the mediation hypothesis (H8) we followed the recommendation of Nitzl et al. (2016) evaluate the indirect effect of gamification on problem-solving learning outcomes. We used the bootstrapping procedure that determines the properties of an estimator such as the variance. The bootstrapping of the sampling distribution shows that the indirect effect is significant (p = 0.043 < 0.05) (H8). This means that emotional engagement does indeed mediate the effect of gamification on problem-solving outcome. This represents an indirect-only fill mediation since the direct effect from gamification to problem solving outcomes is insignificant (as reported above).

To analyze the general quality and power of our model, the explained variance R² and its impact can be used (Hair et al., 2014). The explained variance of the main construct of problem solving can be described as weak (R² < 0.25) (Hair et al., 2010), which is typical for explaining problem-solving outcomes when considering experimental research in management education and online trainings (Janson et al., 2020). Other than this, emotional engagement can explain more than 45% of the variance (R² = 0.455).

Effect sizes tell us the strength of a relationship and assist in evaluating the inner model quality (Hair et al., 2014). Effect sizes were calculated for the determinants of problem solving and emotional engagement. As the direct relationship between gamification and problem solving is not significant (H1), we analyzed the relationship between emotional engagement and problem solving (H5). The effect size f² constitutes the influence of an exogenous construct on endogenous constructs by considering the changes of the coefficient of determination—R² (Cohen, 1988)—where values above 0.02, 0.15, and 0.35 indicate a low, moderate, and high effect, respectively, on the structural level (Henseler et al., 2009). Our results indicate that the effects of emotional engagement on problem solving (H5) can be considered low (f² = 0.127). The effect sizes between learning process satisfaction and emotional engagement (H4) can be considered medium (f² = 0.299), as with the effect sizes between gamification and emotional engagement (f² = 0.152) (H2).

In the last step, we evaluated the predictive relevance as a conclusive assessment of the structural model (Chin, 1998) by using the blindfolding procedure proposed by Stone (1974) and Geisser (1975), which omits one part of the data systematically and uses the resulting estimates to predict the omitted part (Hair et al., 2014). Generally speaking, blindfolding is a sample re-use technique that systematically deletes data points and provides a prognosis on their original values (Hair et al., 2014). We chose d = 7 as the omission distance, which is not a multiple integer of the analyzed cases (n = 68) (Hair et al., 2014). We assessed Q² as the cross-validated redundancy measure to estimate the structural model and measurement models for the data prediction (Hair et al., 2014). The blindfolding procedure is applied to endogenous reflective constructs (Hair et al., 2014). If the value of Q² is larger than 0 for a particular construct, its explanatory variables have a predictive relevance (Henseler et al., 2009). This is the case for both of our constructs of emotional engagement (Q² = 0.359), and learning process satisfaction (Q² = 0.040).

The relative impact of the predictive relevance can be evaluated by the measure q²: values above 0.02, 0.15, and 0.35 respectively indicate a small, medium, or large predictive relevance of constructs, explaining the endogenous construct that is evaluated (Henseler et al., 2009). Our results reflect a medium effect for the relationship between gamification and emotional engagement (H2; q² = 0.152) and the relationship between learning process satisfaction and emotional engagement (H4; q² = 0.299).

Discussion and Implications

Discussion of the Results

In our study, we aimed to understand how the gamification elements of points and badges can support emotional engagement to train students to improve problem-solving outcomes in management education (RQ). In general, the impact of gamification on learning outcomes has often been questioned (Kalogiannakis et al., 2021), resulting in mixed findings about its effectiveness (Bai et al., 2020). To better understand if and how gamification elements can impact specific kinds of outcomes, Koivisto and Hamari (2019) conducted an extensive literature review among all disciplines in gamification research. Out of the 273 empirical studies that were identified, only 12 analyzed engagement and only 8 analyzed satisfaction; moreover, only a few studies used an experimental approach. Thus, scholars have called for the need to conduct more experimental research to analyze gamification (Hammedi et al., 2021), especially by adding control groups to the experiments (Koivisto & Hamari, 2019). Our experimental analysis has shed some light on this research gap.

In our study, we found no positive direct effects of gamification on problem-solving outcomes (H1). Although this hypothesis was rejected, this might be one of the most interesting findings of our study. Many studies have evaluated the positive effects of gamification on learners’ outcomes (De-Marcos et al., 2014; Hew et al., 2016). Consistent with other research that fails to identify the direct effects on learning or behavior outcomes (Super et al., 2019), our study results demonstrate that including points and badges in online training is not a guarantee that management students will be better supported in improving their problem-solving skills. Based on these findings, we suggest that perhaps rather than thinking of gamification as having a direct effect on problem-solving outcomes it is time to think of it as a distal antecedent of problem solving. Thus, we should be focusing on identifying the more proximal explanatory mechanisms that account for why gamification can and should influence problem-solving outcomes.

In this research, we demonstrate that emotional engagement is one such driver of positive problem-solving outcomes from gamification training (H8). When creating online training programs, it is essential to consider how to engage students and to think about the design of the learning environment to increase emotional engagement. We identify that one way to increase emotional engagement is to ensure that training is fair, efficient, clear, and coordinated, such that students are satisfied with the learning process. Thus, based on our results, gamification is successful at developing learner problem-solving outcomes because the learner is satisfied with the process and in turn finds the training enjoyable and useful. Focusing our analyses on emotional engagement enables us to understand why points and badges may result in stronger problem-solving skills for management students.

With our analysis, we aimed to obtain a better understanding of how the relationship between gamification and emotional engagement is moderated by intrinsic and extrinsic goal orientations. In our model, we did not detect any moderating effects of intrinsic goal orientation and extrinsic goal orientation, and had to reject H6 and H7. However, with a negative path coefficient, it is possible that extrinsic goal orientation may harm the positive relationship of gamification and emotional engagement in some scenarios. From this perspective, the goal of striving to obtain for example monetary rewards can weaken the power of gamification on supporting emotional engagement, because gamification is in bigger parts designed to support the inner needs of users such as autonomy (Ryan & Deci, 2006). Our gamification concept is grounded on rewarding learning progress, not by connecting it to correct answers given on knowledge-based tests but by supporting learners to continue their learning process. Our reward concept leads to a positive relationship between gamification and emotional engagement. With learners having a strong extrinsic goal orientation, this positive relationship can be negatively harmed, making the effects of gamification on emotional engagement less effective. With a strong intrinsic goal orientation, it is possible that in some situations the relationship between gamification and emotional engagement would be enhanced by intrinsic goal orientation. However, further analyses need to be done to explain these effects in terms of its significance. It could be interesting to observe if a strong extrinsic goal orientation using a reward-based gamification approach could positively support the relationship of gamification on emotional engagement. Such an approach could be useful for lengthier management courses, as learners could get exhausted more easily.

Implications for Management Education

A challenge in online management learning is to prepare management students to be capable of solving complex business problems once they have graduated. Learning about complex management problems requires educators to create guided and feedback-oriented learning processes (Bigelow, 2004). Regarding management training, educators, and managers should consider gamification as a tool to support learners’ emotional engagement and satisfaction, not by relying on elements such as points or badges but by constructing and designing a meaningful learning experience that learners enjoy and that simultaneously signals their’ achievements to better support them in regulating their online learning experience. A fun and engaging learning experience is also important because, in online learning, it is difficult to provide learners with appropriate feedback about their learning success (Kearns, 2012; Kim et al., 2005). By using elements such as badges or points that signalize the learning progress (e.g., by highlighting how much learners have completed so far), educators can assist learners in improving their learning process by supporting them in continuing with their online training and by rewarding them based on their progress—instead of rewarding or punishing their success or failure according to the results of knowledge-based tests, which deliver negative effects on learning outcomes (Denny, 2013; Haaranen et al., 2014).

To make gamification meaningful, management educators should carefully consider the needs and interests of their management students and the specific contexts instead of randomly selecting and combining elements. For example, research suggests analyzing the conditions under which gamification works best and avoiding pre-existing patterns and designs of the elements (Bai et al., 2020; Dichev et al., 2020). Therefore, educators wishing to support their students in online trainings should carefully consider the elements they select for their training programs; in turn, this can determine the success of a gamification concept (Fogel, 2015; Liu et al., 2017). Educators should also try to determine whether progress or achievements need to be supported. In line with this, creators of online training programs should develop their gamification concepts for training higher-level LGs and problem-solving skills by selecting gamification elements that will better support learners in engaging in their learning processes. Lastly, it is recommended that the factors that support emotional engagement be considered in order to assist learners in better regulating their learning experience.

Implications for Future Research

Following recommendations provided by Brutus et al. (2013), we discussed the internal validity, external validity, construct validity, and statistical conclusion validity for both research implications and future research.

To support internal validity, future studies could investigate different dynamics to analyze how effective they are for the constructs of our research model. In addition to points and badges, other elements, such as rankings or virtual goods, could be considered. Each gamification element enables a different dynamic; for example, ranking stirs competition whereas virtual goods encourage collaborative behavior in learning. Research should also analyze behavioral engagement and cognitive engagement to measure whether there are any effects of gamification. Both kinds of dynamics could be analyzed in future research to better understand the effects of gamification elements that support competition or collaboration (Santhanam et al., 2016).

External validity can be supported by considering different management-related topics and learners with different educational backgrounds, such as managers attending MBA courses online (Arbaugh, 2005). To support construct validity, and to better understand how constructs, such as engagement, goal orientation, and satisfaction, work with the learning process, future research could consider a more detailed and focused analysis of those constructs. Experiments can be used to analyze each construct in more detail. For example, regarding goal orientation, research should try to consider which kind of goal orientation supports learning performance and which designs of gamification concepts are more suitable to address intrinsic goal orientation and extrinsic goal orientation. This could also elicit research about tailored gamification concepts, such as supporting specific kinds of goal orientation (Klock et al., 2020). In terms of the underlying theory, (digital) nudging from behavioral economics (Thaler & Sunstein, 2009; Weinmann et al., 2016) could provide further grounding for inducing behavioral change related to learning through soft-paternalistic mechanisms. This is also true when considering the embedding of gamification into the scaffolding of learning processes to foster learning outcomes in a systematic way (Janson et al., 2020). Lastly, in support of statistical conclusion validity, we suggest future research undertaking a longitudinal analysis with a larger sample size to better understand the long-term effects of gamification on management education.

Limitations

Our study has some limitations. In terms of internal validity, we cannot rule out the possibility of the long-term effects of the gamification elements used in online learning environments. Our results highlight that emotional engagement mediates the relationship between gamification and problem-solving outcomes. However, these results can change over time, leading to findings other than those we saw in our one-time analysis. The fact that our study only examines how management students react toward gamification in the context of online education is likely to affect the generalizability of our results, limiting them to a subset of the target group. With our one-time measurement of emotional engagement and satisfaction for problem-solving outcomes, we cannot interpret the long-term effects of gamification on problem-solving outcomes. Lastly, our statistical measures resulted in three rejected hypotheses. The fact that our study only included a limited number of participants implies that the non-significant effects can change with a larger sample size.

Conclusion

Supporting management students in training their problem-solving skills still needs to be explored in more detail. Companies often lament that new management employees lack the skills to solve the complex problems of daily business. With our study, we demonstrate how teachers and trainers can construct gamified online training that supports management trainees in better emotionally engaging in their learning and thereby strengthening their problem-solving skills. The results of our study demonstrate the importance of emotional engagement in online learning. In the present study, we demonstrate that relying on points and badges is not enough to support the training of problem-solving skills. We can demonstrate that emotional engagement is an important construct to support a positive impact on problem-solving outcomes. We support practitioners in deciding how to design their online teaching courses by using points and badges as one mechanism (among others) to bolster the emotional engagement, and satisfaction with the learning process of their students.

Footnotes

Appendix

Appendix 1.

Measurement Instruments.

Construct	Items	Scale	Reference
Intrinsic goal orientation: A goal refers to a certain end state that a person desires or expects to gain. Such goal orientation can be driven by internal reasons such as curiosity or the positive feeling of being challenged. Being intrinsically driven results in completing achievements in learning supporting a feeling of being capable of solving complex problems alone.
IM1	In a training like this, I prefer course material that really challenges me, so I can learn new things.	7-point Likert scale (strongly disagree to strongly agree)	Pintrich et al. (1991)
IM2	In a training like this, I prefer course material that arouses my curiosity, even if it is difficult to learn.
IM3	The most satisfying thing for me in this training is trying to understand the content as thoroughly as possible.
IM4	When I have the opportunity in this training, I chose course assignments that I can learn from even if I can’t answer all questions correctly.
Extrinsic goal orientation: Extrinsic goal orientation concern the degree to which the student perceives herself to be participating in a task for reasons such as grades, rewards, performance, evaluation by others, and competition.
EM1	Getting many correct answers in this training is the most satisfying thing for me right now.	7-point Likert scale (strongly disagree to strongly agree)	Pintrich et al. (1991)
EM2	The most important thing for me right now is getting as many correct answers as possible.
EM3	I want to get more correct answers in this training than most of the other students.
EM4	I want to do well in this training because it is important to show my ability to my family, friends, employer, and others.
Engagement: Engagement is about enjoying a learning activity, and feeling entertained helping learners to keep engaged in what they are doing.
E1*	~~During the training I was concentrating~~.	5-point Likert scale (strongly disagree to strongly agree)	Adapted from Hamari et al. (2016)
E2*	~~The provided content focused my attention~~.
E3	I enjoyed the training.
E4	The interaction with the training program was entertaining.
E5	The interaction with the training was fun.
Satisfaction with learning process: Learning process satisfaction describes the impact of a processes which has taken place during the learning sessions participated by learners. In a satisfying learning environment, learners act efficiently, and coordinated. Learners understand what they are doing and have the feeling of being fairly treated.
How would you describe your learning process on the scale below?
Satis1	Efficient vs. Inefficient	Bipolar (e.g., efficient vs. inefficient etc.)	Gupta and Bostrom (2013)
Satis2	Coordinated vs. Uncoordinated
Satis3	Fair vs. Unfair
Satis4*	~~Satisfying vs. Dissatisfying~~
Satis5	Understandable vs. Confusing
Control variable
Technology readiness: The behavior process behind the adoption of technological products and services.
TR1	I feel confident doing business with a place that can only be reached online.	7-point Likert scale (strongly disagree to strongly agree)	Adapted from van der Rhee et al. (2007)
TR2	I can usually figure out new high-tech products and services without help from others.
TR3	Technology gives people more control over their daily lives.
TR4	In general, I am among the first in my circle of friends to acquire new technology when it appears.
TR5	Technology makes me more efficient in my occupation.

Note that marked items were removed from the model because their loading was below 0.7.

Acknowledgements

We express our gratitude to the students of the University of Kassel who took part in this study. We would also like to thank Jennifer Hopp for her support in the development of the training and data collection. Furthermore, this research builds on a paper that has been presented at the Academy of Management Annual Meeting 2019 in Boston (Schöbel et al., 2019a). In 2019, this paper was also presented at the Management Education and Learning Writers Workshop at the Academy of Management Annual Meeting 2019. We thank the reviewers and attendees as well as the mentors of the Management Education and Learning Writers Workshop for their valuable feedback that helped us to improve our research and to write this paper. Last but not least, we thank our Associate Editor for her guidance as well as the anonymous reviewers for their constructive feedback and openness during the review process.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: The second author acknowledges support from the basic research fund of the University of St. Gallen. This research received funding by Swiss National Science Foundation (100013_192718).

ORCID iDs

Sofia Marlena Schöbel

Andreas Janson

Notes

References

Anderson

L. W.

Krathwohl

D. R.

(2016). A taxonomy for learning, teaching and assessing: A revision of Bloom’s taxonomy. LongmanPublishing.

Aparicio

Oliveira

Bacao

Painho

(2019). Gamification: A key determinant of massive open online course (MOOC) success. Information Management, 56(1), 39–54.

Arbaugh

J. B.

(2000). How classroom environment and student engagement affect learning in Internet-based MBA courses. Business Communication Quarterly, 63(4), 9–26.

Arbaugh

J. B.

(2005). Is there an optimal design for on-line MBA courses? Academy of Management Learning and Education, 4(2), 135–149.

Bagozzi

R. P.

(1988). On the evaluation of structural equation models. Journal of the Academy of Marketing Science, 16(1), 74–94.

Bai

Hew

K. F.

Huang

(2020). Does gamification improve student learning outcome? Evidence from a meta-analysis and synthesis of qualitative data in educational contexts. Review of Educational Research, 30(7521), 100322. https://doi.org/10.1016/j.edurev.2020.100322

Bigelow

J. D.

(2004). Using problem-based learning to develop skills in solving unstructured problems. Journal of Management Education, 28(5), 591–609.

Blohm

Leimeister

J. M.

(2013). Gamification: Design of IT-based enhancing services for motivational support and behavioral change. Business & Information Systems Engineering, 5(4), 275–278.

Bloom

B. S.

Engelhart

M. D.

Furst

E. J.

Hill

W. H.

Krathwohl

D. R.

(1956). Taxonomy of educational objectives, handbook I: The cognitive domain (Vol. 19). David McKay Co Inc.

10.

Bowman

(2019). Configurations of management cognition and action in a complex world. Academy of Management Proceedings, Briarcliff Manor, NY, USA.

11.

Brown

K. G.

(2005). An examination of the structure and nomological network of trainee reactions: A closer look at “smile sheets”. Journal of Applied Psychology, 90(5), 991–1001.

12.

Brutus

Aguinis

Wassmer

(2013). Self-reported limitations and future directions in scholarly reports: Analysis and recommendations. Journal of Management, 39(1), 48–75.

13.

Bughin

Hazan

Lund

Dählström

Wiesinger

Subramaniam

(2018). Skill shift: Automation and the future of the workforce. https://www.mckinsey.com/featured-insights/future-of-work/skill-shift-automation-and-the-future-of-the-workforce

14.

Buil

Catalán

Martínez

(2020). Engagement in business simulation games: A self-system model of motivational development. British Journal of Educational Technology, 51(1), 297–311.

15.

Carini

R. M.

Kuh

G. D.

Klein

S. P.

(2006). Student engagement and student learning: Testing the linkages. Research in Higher Education, 47(1), 1–32.

16.

Carless

Boud

(2018). The development of student feedback literacy: Enabling uptake of feedback. Assessment & Evaluation in Higher Education, 43(8), 1315–1325.

17.

Chin

W. W.

(1998). The partial least squares approach to structural equation modeling. Modern Methods for Business Research, 295(2), 295–336.

18.

Cohen

(1988). Statistical power analysis for the behavioural sciences. Routledge.

19.

Davis

Singh

(2015). Digital badges in afterschool learning: Documenting the perspectives and experiences of students and educators. Computers & Education, 88, 72–83.

20.

De-Marcos

Domínguez

Saenz-de-Navarrete

Pagés

(2014). An empirical study comparing gamification and social networking on e-learning. Computers & Education, 75, 82–91.

21.

Denny

(2013). The effect of virtual achievements on student engagement. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, Paris, France.

22.

Deterding

Sicart

Nacke

O'Hara

Dixon

(2011). Gamification. Using game-design elements in non-gaming contexts. Proceedings of the CHI'11 Extended Abstracts on Human Factors in Computing Systems, Vancouver, BC, Canada.

23.

Dias

(2017). Teaching operations research to undergraduate management students: The role of gamification. International Journal of Management in Education, 15(1), 98–111.

24.

Dichev

Dicheva

Irwin

(2020). Gamifying learning for learners. International Journal of Educational Technology in Higher Education, 17(1), 1–14. https://doi.org/10.1186/s41239-020-00231-0

25.

Ding

Kim

Orey

(2017). Studies of student engagement in gamified online discussions. Computers & Education, 115, 126–142.

26.

Eom

S. B.

(2014). Understanding e-learners’ satisfaction with learning management systems. Bulletin of the IEEE Technical Committee on Learning Technology, 16(2/3), 10–13.

27.

Eom

S. B.

Ashill

(2016). The determinants of students’ perceived learning outcomes and satisfaction in university online education: An update. Decision Sciences Journal of Innovative Education, 14(2), 185–215.

28.

Eseryel

Law

Ifenthaler

Miller

(2014). An investigation of the interrelationships between motivation, engagement, and complex problem solving in game-based learning. Journal of Educational Technology & Society, 17(1), 42–53.

29.

Ferguson

J. M.

DeFelice

A. E.

(2010). Length of online course and student satisfaction, perceived learning, and academic performance. The International Review of Research in Open and Distributed Learning, 11(2), 73–84.

30.

Fernandes

Duarte

Ribeiro

Farinha

Pereira

J. M.

Silva

M. M. D.

(2012). Ithink: A game-based approach towards improving collaboration and participation in requirement elicitation. Procedia Computer Science, 15, 66–77.

31.

Fisher

C. D.

(2003). Why do lay people believe that satisfaction and performance are correlated? Possible sources of a commonsense theory. Journal of Organizational Behavior, 24(6), 753–777.

32.

Fogel

(2015). Will 80% of gamification projects fail? Giving credit to Gartner’s 2012 gamification forecast. http://www.gameffective.com/gamification-basics/will-80-of-gamification-projects-fail/

33.

Fornell

Larcker

D. F.

(1981). Evaluating structural equation models with unobservable variables and measurement error. Journal of Marketing Research, 18(1), 39–50.

34.

Geisser

(1975). The predictive sample reuse method with applications. Journal of the American Statistical Association, 70(350), 320–328.

35.

Gibbons

A. S.

(2003). What and how do designers design? TechTrends, 47(5), 22–27.

36.

Gibbons

A. S.

Nelson

Richards

(2000). The nature and origin of instructional objects. The Instructional Use of Learning Objects.

37.

Grossman

Salas

Pavlas

Rosen

M. A.

(2013). Using instructional features to enhance demonstration-based training in management education. Academy of Management Learning and Education, 12(2), 219–243.

38.

Gupta

Bostrom

(2013). An investigation of the appropriation of technology-mediated training methods incorporating enactive and collaborative learning. Information Systems Research, 24(2), 454–469. https://doi.org/10.1287/isre.1120.0433

39.

Guzzo

R. A.

(1979). Types of rewards, cognitions, and work motivation. Academy of Management Review, 4(1), 75–86.

40.

Haaranen

Ihantola

Hakulinen

Korhonen

(2014). How (not) to introduce badges to online exercises. Proceedings of the 45th ACM Technical Symposium on Computer Science Education.

41.

Hair

J. F.

Black

W. C.

Babin

B. J.

Anderson

R. E.

(2010). Multivariate data analysis. A global perspective. Pearson.

42.

Hair

J. F.

Hult

G. T. M.

Ringle

Sarstedt

(2016). A primer on partial least squares structural equation modeling (PLS-SEM). SAGE Publications.

43.

Hair

J. F.

Hult

G. T. M.

Ringle

C. M.

Sarstedt

(2014). A primer on partial least squares structural equation modeling (PLS-SEM). SAGE Publications.

44.

Hamari

Koivisto

Sarsa

(2014). Does gamification work? A literature review of empirical studies on gamification. Hawaii International Conference on System Science (HICSS), Waikoloa, HI, USA.

45.

Hamari

Shernoff

D. J.

Rowe

Coller

Asbell-Clarke

Edwards

(2016). Challenging games help students learn: An empirical study on engagement, flow and immersion in game-based learning. Computers in Human Behavior, 54, 170–179.

46.

Hammedi

Leclercq

Poncin

Alkire (Née Nasr)

. (2021). Uncovering the dark side of gamification at work: Impacts on engagement and well-being. Journal of Business Research, 122(3), 256–269. https://doi.org/10.1016/j.jbusres.2020.08.032

47.

Hanus

M. D.

Fox

(2015). Assessing the effects of gamification in the classroom: A longitudinal study on intrinsic motivation, social comparison, satisfaction, effort and, academic performance. Computers & Education, 80, 152–161.

48.

Henseler

Ringle

C. M.

Sarstedt

(2015). A new criterion for assessing discriminant validity in variance based structural equation modeling. Journal of the Academy of Marketing Science, 43(1), 115–135. https://doi.org/0.1007/s11747-014-0403-8

49.

Henseler

Ringle

C. M.

Sinkovics

R. R.

(2009). The use of partial least squares path modeling in international marketing. In Sinkovics

Ghauri

(Eds.), New challenges to international marketing (pp. 277–319). Emerald Group Publishing Limited.

50.

Hew

K. F.

Huang

Chu

K. W. S.

Chiu

D. K. W.

(2016). Engaging Asian students through game mechanics: Findings from two experiment studies. Computers & Education, 92–93, 221–236.

51.

Hiltbrand

Burke

(2011). How gamification will change business intelligence. Business Intelligence Journal, 16(2), 8–16.

52.

Horton

(2011). E-learning by design. John Wiley & Sons.

53.

Hrbackova

Suchankova

(2016). Self-determination approach to understanding of motivation in students of helping professions. Procedia-Social and Behavioral Sciences, 217, 688–696.

54.

Hui

P. J. H.

Clark

T. H.

Tam

K. Y.

Milton

(2007). Technology-assisted learning: A longitudinal field study of knowledge category, learning effectiveness and satisfaction in language learning. Journal of Computer Assisted Learning, 24(3), 245–259.

55.

Hulland

(1999). Use of partial least squares (PLS) in strategic management research: A review of four recent studies. Strategic Management Journal, 20(2), 195–204.

56.

Jang

(2008). Supporting students’ motivation, engagement, and learning during an uninteresting activity. Journal of Education & Psychology, 100(4), 798–811.

57.

Janson

Sӧllner

Leimeister

J. M.

(2020). Ladders for learning: Is scaffolding the key to teaching problem-solving in technology-mediated learning contexts? Academy of Management Learning and Education, 19(4), 439–468. https://doi.org/10.5465/amle.2018.0078

58.

Jonassen

D. H.

(2000). Toward a design theory of problem solving. Educational Technology Research and Development, 48(4), 63–85.

59.

Kalogiannakis

Papadakis

Zourmpakis

A.-I.

(2021). Gamification in science education. A systematic review of the literature. Education Sciences, 11(1), 22–36. https://doi.org/10.3390/educsci11010022

60.

Kearns

L. R.

(2012). Student assessment in online learning: Challenges and effective practices. Journal of Online Teaching and Learning, 8(3), 198–208.

61.

Keller

J. M.

(1987). Development and use of the ARCS model of instructional design. Journal of Instructional Development, 10(3), 2–10.

62.

Khan

Ahmad

F. H.

Malik

M. M.

(2017). Use of digital game based learning and gamification in secondary school science: The effect on student engagement, learning and gender difference. Education and Information Technologies, 22(6), 2767–2804.

63.

Kim

K.-J.

Liu

Bonk

C. J.

(2005). Online MBA students’ perceptions of online learning: Benefits, challenges, and suggestions. The Internet and Higher Education, 8, 335–344.

64.

Kim

M. C.

Hannafin

M. J.

(2011). Scaffolding problem solving in technology-enhanced learning environments (TELEs): Bridging research and theory with practice. Computers & Education, 56(2), 403–417.

65.

Klock

A. C. T.

Gasparini

Pimenta

M. S.

Hamari

(2020). Tailored gamification: A review of literature. International Journal of Human-Computer Studies, 144(3), 102495. https://doi.org/10.1016/j.ijhcs.2020.102495

66.

Koivisto

Hamari

(2019). The rise of motivational information systems: A review of gamification research. International Journal of Information Management, 45(4), 191–210. https://doi.org/10.1016/j.ijinfomgt.2018.10.013

67.

Kuo

F.-R.

Hwang

G.-J.

Lee

C.-C.

(2012). A hybrid approach to promoting students’ web-based problem-solving competence and learning attitude. Computers & Education, 58(1), 351–364.

68.

Kuo

M.-S.

Chuang

T.-Y.

(2016). How gamification motivates visits and engagement for online academic dissemination – An empirical study. Computers in Human Behavior, 55, 16–27.

69.

Kwak

D.-H.

Deng

Kuem

Kim

(2021). How to achieve goals in digital games? An empirical test of a goal-oriented model in Pokémon GO. Journal of the Association for Information Systems, Forthcoming.

70.

Leemkuil

de Jong

(2012). Adaptive advice in learning with a computer-based knowledge management simulation game. Academy of Management Learning and Education, 11(4), 653–665.

71.

Lee

S. J.

Srinivasan

Trail

Lewis

Lopez

(2011). Examining the relationship among student perception of support, course satisfaction, and learning outcomes in online learning. The Internet and Higher Education, 14(3), 158–163.

72.

Liu

C.-C.

Cheng

Y.-B.

Huang

C.-W.

(2011). The effect of simulation games on the learning of computational problem solving. Computers & Education, 57, 1907–1918.

73.

Liu

Santhanam

Webster

(2017). Toward Meaningful Engagement: A framework for design and research of gamified information systems. MIS Quarterly, 41(4), 1011–1034.

74.

Grossman

Fitzmaurice

(2012). Gamicad: A gamified tutorial system for first time autocad users. Proceedings of the 25th Annual ACM Symposium on User Interface Software and Technology.

75.

Nawrot

Doucet

(2014). Building engagement for MOOC students: Introducing support for time management on online learning platforms. International World Wide Web Conference.

76.

Newmann

F. M.

(1992). Student engagement and achievement in American secondary schools. ERIC.

77.

Nitzl

Roldan

J.L.

Cepeda

(2016). Mediation analysis in partial least squares path modeling. Industrial Management and Data Systems, 1846–1864.

78.

Omar

Bhutta

Kalulu

(2009). Assessment of student outcomes in management information systems online course participation. Information Systems Education Journal, 7, 1–10.

79.

Osterwalder

Pigneur

Bernarda

Smith

(2014). Value proposition design: How to create products and services customers want. John Wiley & Sons.

80.

Özhan

Ş.Ç.

Kocadere

S. A

. (2020). The effects of flow, emotional engagement, and motivation on success in a gamified online learning environment. Journal of Educational Computing Research, 57(8), 2006–2031. https://doi.org/10.1177/0735633118823159

81.

Parent

J. D.

Lovelace

K. J.

(2015). The impact of employee engagement and a positive organizational culture on an individual’s ability to adapt to organization change. Eastern Academy of Management Proceedings.

82.

Peterson

T. O.

(2004). So you’re thinking of trying problem based learning? Three critical success factors for implementation. Journal of Management Education, 28(5), 630–647.

83.

Pintrich

P. R.

De Groot

E. V.

(1990). Motivational and self-regulated learning components of classroom academic performance. Journal of Education & Psychology, 82(1), 33–40.

84.

Pintrich

P. T.

Smith

D. A. F.

Garcia

McKeachie

W. J.

(1991). A manual for the use of the motivated strategies learning questionnaire (MSLQ). The Regents of The University of Michigan.

85.

Podsakoff

P. M.

MacKenzie

S. B.

Lee

J.-Y.

Podsakoff

N. P.

(2003). Common method biases in behavioral research: A critical review of the literature and recommended remedies. Journal of Applied Psychology, 88(5), 879–903.

86.

Rich

B. L.

Lepine

J. A.

Crawford

E. R.

(2010). Job engagement: Antecedents and effects on job performance. Academy of Management Journal, 53, 617–635.

87.

Ryan

R. M.

Deci

E. L.

(2006). Self-regulation and the problem of human autonomy: Does psychology need choice, self-determination, and will? Journal of Personality, 74(6), 1557–1585.

88.

Sailer

Homner

(2020). The gamification of learning: A meta-analysis. Educational Psychology Review, 32, 77–112.

89.

Salas

Wildman

J. L.

Piccolo

R. F.

(2009). Using simulation-based training to enhance management education. Academy of Management Learning and Education, 8(4), 559–573.

90.

Santhanam

Liu

Shen

W. C. M.

(2016). Research note—gamification of technology-mediated training: Not all competitions are the same. Information Systems Research, 27(2), 453–465.

91.

Schöbel

Janson

Hopp

J. C.

Leimeister

J. M

. (2019a). Gamification of online training and its relation to engagement and problem-solving outcomes. Academy of Management Annual Meeting (AOM), Boston, MA, USA.

92.

Schöbel

Janson

Jahn

Kordyaka

Turetken

Djafarova

Saqr

Söllner

Adam

Heiberg Gad

Wesseloh

Leimeister

J. M.

(2019b). A research agenda for the why, what, and how of gamification designs: Outcomes of an ECIS 2019 panel. Communications of the Association for Information Systems, 46, 706–721.

93.

Schöbel

S. M.

Janson

Söllner

(2020). Capturing the complexity of gamification elements: a holistic approach for analysing existing and deriving novel gamification designs. European Journal of Information Systems, 29, 641–668. https://doi.org/10.1080/0960085x.2020.1796531

94.

Schöbel

Saqr

Janson

(2021). Two decades of game concepts in digital learning environments – A bibliometric study and research agenda. Computers & Education, 173, 1–23.

95.

Shernoff

D. J.

(2013). Optimal learning environments to promote student engagement. Springer.

96.

Shute

V. J.

D'Mello

Baker

Cho

Bosch

Ocumpaugh

Ventura

Almeda

(2015). Modeling how incoming knowledge, persistence, affective states, and in-game progress influence student learning from an educational game. Computers & Education, 86, 224–235. https://doi.org/10.1016/j.compedu.2015.08.001

97.

Sicart

(2014). Play matters. MIT Press.

98.

Silpasuwanchai

Shigemasu

Ren

(2016). Developing a comprehensive engagement framework of gamification for reflective learning. Proceedings of the 2016 ACM Conference on Designing Interactive Systems.

99.

Stone

(1974). Cross-validatory choice and assessment of statistical predictions. Journal of the Royal Statistical Society Series B, 36(2), 111–133.

100.

Super

Keller

R. H.

Betts

T. K.

Roach Humphreys

(2019). Simulation games: Learning goal orientations and norms for knowledge sharing. Academy of Management Proceedings.

101.

Swain

Kumlien

Bond

(2020). An experiential exercise for teaching theories of work motivation: Using a game to teach equity and expectancy theories. Organization Management Journal, 17, 119–132.

102.

Taylor

S. S.

Statler

(2014). Material matters. Journal of Management Education, 38(4), 586–607. https://doi.org/10.1177/1052562913489976

103.

Thaler

R. H.

Sunstein

C. R.

(2009). Nudge: Improving decisions about health, wealth, and happiness. Penguin.

104.

Thiebes

Lins

Basten

(2014). Gamifying information systems: A synthesis of gamification mechanics and dynamics. European Conference on Information Systems (ECIS).

105.

Thurlings

Vermeulen

Bastiaens

Stijnen

(2013). Understanding feedback: A learning theory perspective. Review of Educational Research, 9, 1–15.

106.

Treiblmaier

Putz

L.-M.

Lowry

P. B.

(2018). Research commentary: Setting a definition, context, and theory-based research agenda for the gamification of non-gaming applications. Association for Information Systems Transactions on Human-Computer Interaction (THCI), 10(3), 129–163.

107.

van der Rhee

Verma

Plaschka

G. R.

Kickul

J. R

. (2007). Technology readiness, learning goals, and eLearning: Searching for synergy. Decision Sciences Journal of Innovative Education, 5(1), 127–149.

108.

Wan

Compeau

Haggerty

(2012). The effects of self-regulated learning processes on e-learning outcomes in organizational settings. Journal of Management Information Systems, 29(1), 307–340.

109.

Watson

Sutton

J. M.

(2012). An examination of the effectiveness of case method teaching online: Does the technology matter? Journal of Management Education, 36(6), 802–821.

110.

Weinmann

Schneider

Brocke

J. V.

(2016). Digital nudging. Business & Information Systems Engineering, 58(6), 433–436.

111.

Wood

Bruner

J. S.

Ross

(1976). The role of tutoring in problem solving. Journal of Child Psychology and Psychiatry, 17(2), 89–100.

112.

Zyda

(2005). From visual simulation to virtual reality to games. Computer Society, 38(9), 25–32.