Evaluating the Effectiveness of Game-Based Learning for Long-Term Knowledge Retention

Abstract

This longitudinal study investigated the efficacy of game-based learning in enhancing the retention of safety knowledge, specifically within the domain of construction safety. The research focused on assessing participants’ retention of safety knowledge 1 year after game-based training, revealing a resilient performance despite a perceived decline in preparedness. Analysis of the data indicated no statistically significant variance in participants’ performance in hazard identification tests conducted 1 year after the initial assessment following game-based training. The study also explored the relationship between perceived preparation, actual performance, and the temporal gap between evaluations, highlighting the necessity for a comprehensive understanding of these dynamics in educational settings. The implications of this research extend beyond construction safety, underscoring the transformative potential of game-based learning in cultivating enduring safety awareness across various industries. By harnessing innovative educational technologies, organizations can enhance workplace safety protocols and mitigate occupational hazards.

Introduction

Game-based learning has emerged as a pioneering force in educational technology, offering learners greater engagement, motivation, and active participation (Wouters et al., 2013). This technology has the potential to provide interesting learning environments leading to greater learner engagement (Kanade & Duffy, 2020). Its application spans various domains, from fire safety drills (Mystakidis et al., 2022) to cultural immersion experiences (Gao et al., 2021) and live-line worker simulations (Park et al., 2006). Effective knowledge retention is a crucial metric in evaluating the efficacy of any training method, as it determines the enduring impact on participants’ understanding. Traditionally, conventional training methods like lectures have struggled to ensure knowledge retention among students 3-6 months after training (Hamilton, 2005). One contributing factor to this issue is the passive role of learners in the conventional training approach, lacking active participation in the learning process (Kanade & Duffy, 2022). However, assessing knowledge retention necessitates longitudinal studies spanning extended durations making it a difficult task to accomplish as retaining participants over a longer period can be tricky. It is important to study knowledge retention as time elapsed between training and application of training can lead to decay in knowledge (Kanade & Duffy, 2024). There have been a few studies which study the phenomenon of long-term knowledge retention. Babu et al. (2018) studied the effectiveness of virtual reality training in educating the participants about various automotive parts. It was found that the participants trained using virtual reality exhibited better knowledge recall 1 day after the training compared to those trained using 2D tablet devices (Babu et al., 2018). Similarly, medical students who were trained using virtual reality exhibited better knowledge recall 4 months after training compared to those who were trained using a conventional training method (Ebert & Tutschek, 2019). However, when virtual reality training was compared with traditional training in the context of educating US Navy personnel, there was no difference in the knowledge retained by participants 1 week after the training (Hall et al., 1998). This shows that the effect of various training methods on knowledge retention is context specific.

The construction industry, known for its hazardous working conditions, recorded 1,056 occupational fatalities in the United States alone in 2022 (Bureau of Labor Statistics, 2023). Accidents on construction sites often stem from workers’ inadequate awareness of potential dangers (Jeelani et al., 2020 Construction safety is a critical aspect of the construction industry, necessitating continuous vigilance and strict adherence to safety protocols to mitigate the inherent risks involved in construction activities. The retention of safety knowledge is essential to ensure that workers and stakeholders remain aware of safety procedures, hazards, and best practices over time, thereby reducing the likelihood of accidents and injuries on construction sites. Previous research has demonstrated that a positive safety climate consists of both adherence to safety protocols and constructive responses to errors (Leroy et al., 2012). Additionally, Huang & Yang (2019) have presented applications for knowledge exchange and safety awareness in construction, highlighting the significance of knowledge retention in promoting safety within the industry (Huang & Yang, 2019). Furthermore, Deng et al. (2022) found that safety vigilance has a positive effect on discovering information links, further underlining the importance of continuous vigilance in identifying and addressing potential hazards (Deng et al., 2022). These findings underscore the critical role of retention of safety knowledge and adherence to safety protocols in ensuring construction safety and mitigating risks in construction activities.

In this context, it is important to explore the effectiveness of game-based learning in enhancing knowledge retention. The Occupational Safety and Health Administration has developed a game to help small business owners train their employees in workplace safety. This game has a construction module that was used for training in our previous study. The objective of this study was to evaluate the effectiveness of the OSHA hazard identification tool for long-term knowledge retention. To the best of our knowledge, the efficacy of the OSHA hazard identification tool for long-term knowledge retention has not been tested yet. Building upon our previous research that proved the effectiveness of the OSHA hazard identification tool in enhancing the safety knowledge of the participants, this study sought to evaluate the durability of knowledge gained during the game-based training session among participants. In the present study, a combination of subjective and objective measures was employed to ensure the validity and reliability of the results.

Method

Participants

This study was approved by the Purdue Institutional Review Board (IRB-2022-1519) ensuring the safety of the participants. In the initial study which was conducted to evaluate the effectiveness of game-based learning for construction safety training, 32 graduate students volunteered for participation. Ten participants (23–30years old, mean age 26.2 years, 7 males, 3 females) out of the 32 who volunteered for the initial study signed up for the retention test. The participants were compensated with a $10 gift card for their time.

Design of Experiment

To evaluate the safety knowledge of participants, a hazard identification test was designed which consisted of 10 images of real construction sites being shown to the participants (1 min each) and the participants were asked to identify as many hazards as possible on each image. An example of the image used in the test is shown in Figure 1. The images were presented using a Tobii TX300 eye-tracker. However, analysis of the eye-tracker data collected during the study has not been included in this article. The participants were instructed to speak out about the hazard as soon as they identified it in the image and their voice was recorded. These voice recordings were transcribed using MS Word and then the transcripts were analyzed to calculate the number of hazards identified by the participants. The participants were asked to fill out a NASA-TLX questionnaire and a Qualtrics survey after each hazard identification test to evaluate their mental workload and their perception of preparation. In the initial study, this test was conducted before the training session that is, a pre-training test, to establish a baseline level of the participant’s knowledge, and then after the training session that is, a post-training test to evaluate the change in knowledge level. During the training session, the participants used the construction module of the OSHA hazard identification tool to explore various construction-related hazards in the game environment. The results of the initial study are outside the scope of this article. In the current study, the hazard identification test was conducted 1 year after the initial study to evaluate the knowledge level retained by the participants. The facial expressions of the participants during the hazard identification test were recorded and analyzed, the results of which are beyond the purview of this article.

Figure 1.

An example of the image of a construction site used in the hazard identification test (images courtesy of David Ausmus, with permission).

Dependent Measures

The Hazard Identification Index (HII), which is the ratio of the number of hazards identified on each image to the total number of hazards present on the image, was used as an objective measure to evaluate the safety knowledge of the participants. Five certified safety managers, each with a minimum of a decade’s experience, independently identified hazards in the test images to calculate the total number of hazards. The details of this process can be found in (Hasanzadeh et al., 2017). Participants’ responses to the NASA-TLX questionnaire and the Qualtrics survey were used as subjective measures to evaluate the effectiveness of game-based training. The HII, metrics from the NASA-TLX questionnaire and the results from the Qualtrics survey from the pre-training, post-training and retention test were compared to evaluate differences. The comparisons between the pre-training test and post-training test have not been included in this article.

Results

Comparison of Hazard Identification Index

The average Hazard Identification Index (HII) across 10 images of participants in the retention test was compared with that of the post-training test. A paired t-test was used for the comparison.

H₀: There was no significant difference in the average HII for the post-training and retention tests.

H₁: The average HII for the retention test was significantly lower compared to the post-training test.

The result shows that there was no significant difference in the average HII scores between the post-training test (M = 0.54, SD = 0.10) and the subsequent retention test (M = 0.55, SD = 0.09), t(9) = −1.29, p = .22. This demonstrates the ability of the participants to retain the knowledge acquired during the training session, even after a gap of 1 year preceding the retention test.

Comparison of perception of performance

The responses to the NASATLX questionnaire after the post-training test and the retention test were analyzed. The comparison yielded significant results for the metric related to the perception of performance.

H₀: There was no significant difference in the perception of performance after the post-training and retention tests.

H₁: The perception of performance after the retention test was significantly higher compared to the post-training test.

The result revealed that participants’ perception of their performance after the retention test (M = 11.20, SD = 2.96) was significantly greater compared to the post-training test (M = 7.41, SD = 3.58), t(9) = −3.83, p = .002. A box plot demonstrating the significant increase in the perception of performance among participants after the retention test has been shown in Figure 2.

Figure 2.

Box plot demonstrating the perception of increased performance among participants after the retention test.

Comparison of Perception of Preparation

From the data obtained through the Qualtrics survey, the perceived preparation level of the participants after the post-training test and the retention test was compared.

H₀: There was no significant difference in the perception of preparation after the post-training and retention tests.

H₁: The perception of preparation after the retention test was significantly lower compared to the post-training test.

The result demonstrated that participants reported a significantly lower sense of preparedness after the retention test (M = 5.00, SD = 1.83) in comparison to the post-training test (M = 7.12, SD = 1.67), t(8) = −2.68, p = .013. A box plot demonstrating the significant decrease in the perception of preparation among participants after the retention test has been shown in Figure 3.

Figure 3.

Box plot demonstrating the perception of lower preparation level among participants for the retention test.

These findings indicate a decrease in participants’ perceived preparedness for the hazard identification test as time passed. However, participants believed that their performance in the retention test was better than in the post-training test.

Discussion

This study provides insights into the effectiveness of game-based learning in enhancing safety knowledge retention, particularly in the context of construction safety. The research demonstrates the enduring impact of game-based training, revealing participants’ ability to retain information related to various hazards over an extended period. This is particularly promising in the construction industry, where maintaining a high level of safety awareness is essential to mitigate occupational risks (Mohd et al., 2019). This study also explores the relationship between perceived preparation, actual performance, and the temporal gap between evaluations. Despite participants’ perceived decline in preparedness over time, their performance remained robust during the retention test, indicating the effectiveness of game-based training in helping participants identify hazards 1 year after the training. It is interesting to note that the participants’ self-perception of their performance in the retention test was lower than their self-perception a year prior. This decline in self-perception could be attributed to the participants’ anticipation of a decrease in their level of preparation over the one-year gap, leading to a subsequent decrease in confidence in their hazard identification skill. This highlights the importance of considering learners’ perceptions alongside objective performance metrics when evaluating educational interventions. Learners and educators must be assured of the efficacy of game-based learning in fostering long-term knowledge retention. This assurance will encourage broader acceptance and utilization of this technology. Furthermore, the focus on safety awareness in the construction industry is timely, given the persistent challenges associated with workplace safety in such environments (Mohd et al., 2019). By showcasing the potential of game-based training, the research offers a promising avenue for enhancing safety practices and reducing occupational hazards within the construction industry. The longitudinal approach adopted in this study provides valuable insights into the durability of safety knowledge acquired through game-based training, emphasizing the need for continued investment in innovative educational technologies for sustained improvements in workplace safety.

Conclusion

In conclusion, the study underscores the transformative potential of game-based learning in fostering enduring safety awareness and promoting a culture of safety within workplaces. By shedding light on the effectiveness of game-based training in enhancing safety knowledge retention, the research paves the way for further exploration into factors influencing knowledge retention and optimizing educational interventions. Continued research in this area is crucial for advancing the efficacy of safety training initiatives, not only within the construction industry but across diverse sectors where workplace safety is a priority. Ultimately, leveraging the power of game-based learning can better equip workforces with the knowledge and skills necessary to prevent accidents and ensure a safer working environment for all.

Future Work

While this study has provided valuable insights, it is important to acknowledge its limitation in terms of a small sample size. To address this, future research endeavors could enhance our understanding by examining the efficacy of game-based learning for long-term knowledge retention in real workplace settings, such as construction sites, with larger and more diverse sample sizes. This would allow for a more comprehensive assessment of the generalizability and effectiveness of game-based learning interventions in practical occupational environments. Furthermore, given the prevalence of frequent interruptions, particularly from cell phones, in modern-day workplaces, it would be valuable to investigate the impact of interruptions on workers’ hazard identification abilities. Understanding how interruptions affect hazard identification is crucial for developing strategies to mitigate their negative effects on workplace safety. Furthermore, exploring how game-based training influences learners’ capacity to navigate interruptions in real-world workplace scenarios could also be a promising avenue for further investigation. Another potential direction for future research is to delve deeper into the specific mechanisms through which game-based learning facilitates long-term retention of safety knowledge. By employing neuroscientific methods such as functional magnetic resonance imaging (fMRI) or electroencephalography (EEG), researchers can investigate the neural processes underlying the retrieval of safety-related information following game-based training. Understanding the neurocognitive mechanisms involved in long-term retention can provide valuable insights for optimizing the design and delivery of game-based safety training programs.