Sage Journals: Discover world-class research

Abstract

In recent years Virtual Reality has been revitalized, having gained and lost popularity between the 1960s and 1990s, and is now widely used for entertainment purposes. However, Virtual Reality, along with Mixed Reality and Augmented Reality, has broader application possibilities, thanks to significant advances in technology and accessibility. In the current study, we examined the effectiveness of these new technologies for use in education. We found that learning in both virtual and mixed environments resulted in similar levels of performance to traditional learning. However, participants reported higher levels of engagement in both Virtual Reality and Mixed Reality conditions compared to the traditional learning condition, and higher levels of positive emotions in the Virtual Reality condition. No simulator sickness was found from using either headset, and both headsets scored similarly for system usability and user acceptance of the technology. Virtual Reality, however, did produce a higher sense of presence than Mixed Reality. Overall, the findings suggest that some benefits can be gained from using Virtual and Mixed Realities for education.

Keywords

human-computer interface interactive learning environments simulations virtual reality mixed reality augmented reality technology enhanced learning

Although virtual environments have arguably existed since the 1920s (Rolfe & Staples, 1986), with early head-mounted displays developed in the 1960s (Comeau & Bryan, 1961; Goertz et al., 1965), virtual reality (VR) technology has only just become commonly accessible, driven primarily by the entertainment industry. VR is a computer-generated 3D environment, typically viewed in a headset, which replaces the real world. Sales figures for two of the most popular VR headsets (Oculus Rift and HTC Vive) are estimated to be in the millions. However, other approaches have been introduced in which ‘virtual objects’ are overlaid onto the real world; Augmented Reality (AR) and Mixed Reality (MR) systems. The difference between MR and AR is more complex. Milgram and Kishino introduced the concept of a reality–virtuality continuum (Milgram et al., 1995), which used the term mixed reality as an umbrella term encompassing augmented reality and augmented virtuality. Since then the terms have been used differently, with variations of the reality–virtuality continuum, or mixed reality spectrum. Today MR is more commonly defined as being spatially aware, allowing real and virtual objects to interact, whereas AR does not. MR combines the real and virtual worlds, with physical and digital objects co-existing and interacting in real-. Extended Realities (XR, a term which encapsulates VR, AR and MR) are often treated as the same, but they offer very different experiences and possibilities; therefore, their advantages and disadvantages need to be matched to any given application.

Education is often slow to adopt technological improvements (e.g., Cuban, 1986; Gumport & Chun, 2005; Jacobsen, 2000; Selwyn, 2017). However, learning with electronic media (e-learning) has been shown to be effective in numerous studies (e.g., Rosenberg, 2001; Zhang et al., 2004). In a meta-analysis including 1105 papers, Schmid et al. (2014) found a small but reliable advantage for non-Internet computer-assisted instruction compared with classroom instruction. In a recent study, Allcoat and von Mühlenen (2018) compared learning in a VR condition with traditional and video conditions. They found not only improved learning for VR, but VR participants also reported higher engagement and more positive emotions than those in the other conditions. Being in a positive mood can also have reciprocal effects on learning, for example by enabling increased cognitive flexibility (Nadler et al., 2010), or simply by creating a positive academic climate (Seligman et al., 2009). Indeed, Olmos-Raya et al. (2018) found a significant effect of both positive emotion and high immersion on knowledge acquisition (for a discussion of the benefits of immersion for learning see Dede, 2009). Participants also reported that learning in VR was more engaging than the traditional approach in Bennie et al. (2019).

According to the constructivist learning theory (e.g. Duffy & Jonassen, 2013; Fosnot, 1996) learning is an active process, whereby learners construct knowledge for themselves (as opposed to passively receiving information). This theory builds on ideas and suggestions from Piaget’s theory of cognitive development (e.g. Piaget, 1937, 1950), Dewey’s functional psychology (Dewey, 1938) and Vygotsky’s social development theory (Vygotsky, 1980), which have been expanded and researched within cognitive psychology.

E-learning often promotes active learning through interactive technology tools, which constructivism would claim as beneficial. Kay (2011) developed an evaluation scale for Web-Based Learning Tools (WBLT) which focuses on three key constructs: learning, design, and engagement. Kay found an improvement in pre- vs. post-test scores on remembering, understanding, application, and analysis when using WBLT compared to standard methods of teaching. These categories were derived from the revised Bloom’s Taxonomy (Bloom, 1956; see also Anderson et al., 2001). This is a model used to classify educational learning objectives based on cognitive principles and suggests that there is not simply one way in which information is processed and learnt, instead proposing a hierarchy of learning. This hierarchy consists of six stages of cognitive processing from simplest to most complex: remember, understand, apply, analyse, evaluate, and create.

Technologies can complement classroom teaching rather than replace it, with technologies like VR being particularly applicable for teaching practical tasks, and virtual laboratories can offer advantages over traditional methods, such as providing greater flexibility for conducting experiments (Valdez et al., 2015; for reviews see Albidewi & Tulb, 2014; Hilgarth, 2010; Welsh et al., 2003). Similarly, Pan et al. (2006) noted that the use of XR could help enhance, motivate and stimulate learners’ understanding, as well as improve their overall mood. Simulations can be used to mimic real experiments which are important in education for many subjects (e.g., Davies, 2008), and understanding has been shown to be equivalent for both physical and virtual experiments (Zacharia & Olympiou, 2011). Experiments and practicals are often important interactive learning tools, and XRs can be beneficial when physical experiments are not practical. Indeed, interactivity and feedback enhance learning by promoting active rather than passive learning. Active learners are more engaged, motivated, and show better learning than passive learners (Benware & Deci, 1984), leading to better student outcomes (Chi & Wylie, 2014; Cui, 2013).

Markant and Gureckis (2014) noted that according to one explanation active learning improves performance by enhancing cognitive processes related to motivation, attention, and engagement (see also Chi & Wylie, 2014). However, they theorised that the difference between active learning and passive learning comes from a hypothesis-dependent sampling bias which happens when an individual collects data to test their own hypotheses. This explanation is in line with constructivism, which also focuses on the importance of the interaction between experiences and ideas. Thus VR and MR, which enable active learning, allow one to experience and test such learner-generated hypotheses, which may be more effective compared to traditional passive learning methods (e.g. lectures, textbooks).

Freeman et al. (2014) conduced a meta-analysis of 225 studies which compared student performance in undergraduate science, technology, engineering, and mathematics (STEM) courses under traditional lecturing versus active learning conditions. They found that students in classes with traditional lecturing performed worse in the examination and were 1.5 times more likely to fail than students in classes with active learning methods. Similarly, another meta-analysis also found a benefit of using computer simulations for STEM learning (D’Angelo et al., 2014). Technology, including XR, can be a useful tool to enhance learning and engagement in engineering, as discussed by Koretsky and Magana (2019), which is the subject area this study uses.

Active learning via the use of XR provides a variety of benefits. However, the costs and benefits of different XRs (VR vs. MR) might differ, as might user acceptance, engagement and experience. For example, VR presents a closed world, so participants might feel more concealed and private, and more comfortable with the simulated reality, which could, in turn, lead to higher user acceptance. Conversely, participants might feel more vulnerable due to being unaware of their real-life surroundings. Similarly, there are two alternatives for MR; participants might feel more comfortable because they can see their surroundings and what is going on around them, or less comfortable because of their awareness of people who can see them using the technology.

This user experience will be in part linked to the level of immersion and presence generated by the equipment (see Cheng & D’Angelo, 2018). User-experience and user comfort are important considerations in this study, as even if a technology has a variety of benefits, if individuals are unwilling to use it due to discomfort, it will not be useful. As such, simulator sickness (i.e., motion sickness caused by simulated environments) is also an important factor that needs to be considered. Antoniou et al. (2017) discuss the importance of applying good design principles to XR use for education, rather than relying on the novelty effect. They recommend using all spatial and overall environmental degrees of freedom in order to maintain not only the immersion and engagement but also the educational impact, and they suggest incorporating game design principles.

Although previous work has shown the benefits of VR-based learning over traditional methods (for a review see Kavanagh et al., 2017), very little previous work has considered the benefits and costs of learning in MR environments. Accordingly, in the present work, we directly compared learning, user experience, engagement and acceptability in a learning context. Overall, based on constructivist principles, we would expect VR and MR to produce better learning outcomes than traditional methods. Furthermore, we would expect VR and MR to result in improved user experience (e.g., in terms of emotion, engagement, and usability) compared with traditional methods (Allcoat & von Mühlenen, 2018). Finally, we would expect that because VR is fully immersive, it would lead to a higher sense of presence, but it might also lead to more simulator sickness, compared to MR.

Method

Participants

Seventy-five participants (34 female, mean age 25 years) were recruited from the University of Warwick participant pool. Of these, 25 were undergraduate students, 12 postgraduate students, 23 PhD students, and 15 staff members. Fourteen participants had a background in Engineering, 18 in Science, 16 in Economics/Business, and 27 in other subject areas. All reported normal or corrected-to-normal vision and received £5 Amazon vouchers for their participation. Each gave informed written consent and the study was approved by the University of Warwick Humanities & Social Sciences Research Ethics Committee. Participants were randomly assigned to one of three conditions: traditional, VR, and MR and did not differ in terms of self-reported computer skills, gaming experience, or VR/MR headset experience (see Table 1).¹

Table 1.

Demographics and Technology Experience (0–5 Scale) of Each Condition With Results of Comparisons (1-Way ANOVA) Between Conditions.

Measure	Traditional	VR	MR	F(2,72)	p-value
Characteristics
N(M/F)^a	25 (10/15)	25 (11/14)	25 (13/12)		–
Age (years)	23.8 (5.2)	27.5 (9.9)	23.7 (4.8)	2.37	.101
Prior experience
Computer	3.0 (1.2)	3.5 (0.7)	3.5 (0.8)	2.18	.120
Gaming	0.9 (1.0)	0.9 (1.2)	0.9 (1.3)	0.02	.979
VR headset	0.7 (0.9)	1.0 (0.8)	0.8 (0.9)	0.96	.388

Note. ^aAll values except for N (M/F) are given as means (with standard deviations).

Apparatus

The questionnaires and learning material in the traditional condition were presented on a 19” LCD computer screen (1920 x 1080 pixels, 60 Hz) via Microsoft PowerPoint and Qualtrics. Responses were collected through mouse and keyboard. The VR condition used an HTC Vive, which displays a 3D environment via two OLED displays (1080 x 1200 pixels per eye 90 Hz) with a field of view of 100 x 110 degrees. The MR condition used a Microsoft HoloLens, which projects 3D objects on a pair of translucent screens (1268 x 720 pixels per eye, 60 Hz) with a projection field of 30 × 17 degrees (i.e., the 3D objects are seen overlaying the real world). The headsets were of similar weight (Vive 550g, HoloLens 557g) and navigation occurred using the standard handheld controllers. In the XR conditions, participants were tested individually due to the space requirements of the headsets. In the traditional condition, groups of up to eight participants were tested in a computer-equipped teaching lab, providing a traditional learning environment.

Learning Material

The learning material was based on a real classroom example covering solar panels for Engineering students. The VR and MR simulations took the material from an existing course that would usually be taught via PowerPoint slides and developed it to be presented in an immersive 3D environment. This topic focused on students’ understanding of how different parameters can influence solar-power panel efficiency, such as light intensity, panel mismatching and tilt angle. The existing method of delivering these lectures is to use slides showing graphs describing the relationship between solar-power efficiency and other system parameters. The Traditional learning condition was implemented in the form of lecture slides adapted from the course, which participants could navigate through, as in a distance learning environment.

The simulations for VR and MR were created and programmed in Unity, with some models made with the Blender software package. Both the VR (Figure 1a) and MR (Figure 1b) conditions allowed participants to interact with the application to experiment with how different characteristics (e.g., type of solar panel, light intensity, shading) impact power output. A video showing the VR and MR environments can be found at https://youtu.be/Jg3gsjVYrKM.

Figure 1.

Screenshots of the learning environments. (a) VR and (b) MR. Controls for adjusting a number of solar panel variables can be seen in both, along with a diagram showing electron and current flow in solar cells.

Participants were instructed to interact with the learning environment to find out how solar panels work, so they could answer questions in a subsequent test. They were told verbally how to use the equipment and how to navigate the learning environment. Buttons and sliders manipulated variables, and participants could select information boxes to obtain further details. Both the HTC Vive and Microsoft HoloLens have full head-tracking, so participants were able to look around at the 3D environment/objects at will.

The researchers worked closely with the course lecturer to ensure each condition was equivalent in terms of its content, material, and amount of information presented. Both the VR and MR conditions presented the same models and written information. No audio was used to reduce confounding variables.

Procedure and Design

There were three phases: pre-test, learning, and post-test (see Table 2). Participants had approximately 10 minutes for phase 1, at the end of which they put the headset on or were given the lecture slides for the traditional condition. They were given 10 minutes with the learning materials (as determined by piloting²) before they filled in the questionnaires of the post-test. Some questionnaires were only presented once, whereas others were given before and after learning, leading to a mixed design with the between-subject factor Condition (traditional, VR, and MR) and the within-subject factor Test (pre and post).

Table 2.

Characteristics of the Sample With Internal Reliability Estimates Based on Cronbach’s α.

Measure/scale/questions	Scale	NQ^a	Conditions	α
Phase 1 (10 min)
Technology experience questions	0–5	6	All	–
Knowledge test I	–	8	All	–
Differential Emotions Scale (DES) I	1–5	10	All	–^b
Simulator Sickness Questionnaire I	0–3	16	All	.84
Phase 2 (10 min)
Learning materials (study period)	–	–	All
Phase 3 (10–15 min)
Knowledge test II	–	8	All	–
Differential Emotions Scale (DES) II	1–5	10	All	–
Simulator Sickness Questionnaire II	0–3	16	All	.82
Web-Based Learning Tools (WBLT) Evaluation Scale	1–5	13	All	.95
Perceived Quality Scale	1–5	18	VR/MR	.96
System Usability Scale	1–5	10	VR/MR	.90
Unified Theory of Acceptance and Usage of Technology (UTAUT) Questionnaire	1–7	23	VR/MR	.90
Igroup Presence Questionnaire (IPQ)	1–7	14	VR/MR	.83

Note. ^aNQ: number of questions.

^bFor a DES reliability measure, see Boyle (1984).

Knowledge Test

Participant's knowledge of the learning material was assessed using eight questions constructed by the lecturer, based on those used in a real classroom course (see Online Appendix). Participants completed the test twice, once before and once after the learning phase. The questions were a mix of formats and tested different types of knowledge in accordance with Bloom’s Taxonomy (Bloom, 1956): four multiple-choice questions tested ‘remembering’ aspects, three short-answer questions and one calculation question focused on the ‘understanding’ and ‘applying’ aspects (see also Anderson et al., 2001). All questions were marked as correct or incorrect using a marking scheme provided by the lecturer.

User Experience

The Differential Emotions Scale (DES, Izard et al., 1974) was used to measure participants’ emotions before and after engaging with the learning materials. The scale included ten emotion categories (interest, amusement, sadness, anger, fear, anxiety, disgust, contempt, surprise, and elatedness), each represented with three words (e.g. surprised, amazed, astonished). Participants indicated on a five-point scale (from “not at all” to “very strongly”) the extent to which these adjectives corresponded to their current emotional state.

Participants completed Kennedy et al.’s (1993) Simulator Sickness Questionnaire, which assesses to what extent individuals experience physical discomfort. This questionnaire has participants rate whether any of 16 symptoms (e.g., nausea, headache) are affecting them on a four-point scale (from “none” to “severe”).

Student engagement was measured via the WBLT Evaluation Scale (Kay, 2011), developed specifically for evaluating the efficacy of web-based learning tools for education. It consists of 13 questions split into three sections which ask participants to rate on a five-point scale (from “strongly disagree” to “strongly agree”) how well they could learn from the learning tools (learning), how well designed the tools were (design), and how engaging they found them (engagement).

Technology Evaluation

The quality of the learning materials was assessed via the Perceived Quality Scale (Pribeanu et al., 2017) specifically developed for the evaluation of AR-based learning applications. It consists of 18 questions which measure participants’ perceptions of the quality of the learning materials on a five-point scale (from “strongly disagree” to “strongly agree”). Quality was further split into three different sub-scales: ergonomic quality (perceived learnability and ease-of-use), learning quality (perceived efficiency and usefulness), and hedonic quality (cognitive absorption and perceived enjoyment). Minor changes to the wording of the questions were made to fit the context of the scenario.

The System Usability Scale (Brooke, 1996) consists of ten questions which measure the usability of the learning environment on a five-point scale (from “strongly disagree” to “strongly agree”). Example questions include: “I found the system unnecessarily complex”, and “I felt very confident using the system”.

The Unified Theory of Acceptance and Usage of Technology Questionnaire (Venkatesh et al., 2003; Akbar, 2013) was used to measure user acceptance and comfort with being in a 3D simulated environment. 23 of the 31 questions relevant to the scenario of the current study were used (e.g. “Using the system will enable me to accomplish tasks more quickly”). Ratings were provided on a seven-point scale (from “fully disagree” to “fully agree”).

Finally, sense of presence is an aspect of immersion which can impact learners in 3D virtual worlds (Mount et al., 2009, see also McMahan, 2003) was measured with the Igroup Presence Questionnaire (Schubert et al., 2001). This scale has 14 questions (e.g., “I felt present in the virtual space”) rated on a seven-point scale (from “fully disagree” to “fully agree”).

Results

Knowledge Test

Questions in the knowledge test were marked as correct or incorrect to give a total score of 0 to 8. Learning was represented by the difference between the knowledge pre-test and post-test scores (see Table 3). The data from five participants who scored very high in the pre-test (2SD above the mean, i.e., ≥75%) were subsequently removed as outliers from the learning data (Tukey, 1977).

Table 3.

Knowledge Test Means (With Standard Deviations) with Results of Comparisons (1-Way ANOVA) Between Conditions.

Measure	Traditional	VR	MR	F(2,67)	p-value
N	25	25	20		–
Pre-test	1.80 (1.3)	1.96 (1.4)	2.25 (1.5)	0.81	.45
Post-test	4.48 (1.1)	5.30 (1.8)	4.45 (1.7)	1.39	.26
Difference	2.68 (1.2)	3.24 (2.1)	2.20 (1.8)	2.13	.13

A mixed-design ANOVA with the between-subject factor of Condition (traditional, virtual, mixed) and the within-subject factor of Test (pre, post) revealed a significant main effect of Test, F(1,67) = 177.72, p = .001, η_p² = .726; participants’ knowledge improved on average by 2.5 points from pre- to post-test. There was a trend for more learning in the VR condition and less learning in the MR condition, however, this difference did not reach significance, 2-way interaction, F(2,67) = 2.13, p = .13, η_p² = .060. Further analysis of the knowledge data showed that the amount of learning did not depend on prior computer skills (correlation with learning across conditions: r = -.08, p = 51) or gaming skills (r = .22, p = .07), nor did it depend on the amount of previous headset experience (r = .14, p = 25).

Emotional and Physical Experience

The ten pre- and post-test DES questions (see Table 4) were analysed separately for positive emotions (interest, amusement, surprise, and elatedness) and negative emotions (sadness, anger, fear, anxiety, disgust and contempt). A mixed-design ANOVA for the positive emotions with the between-subject factor Condition (traditional, VR, and MR) and the within-subject factors Test (pre and post) and Emotion (interest, amusement, surprise, and elatedness) revealed a significant three-way interaction, F(6, 216) = 2.42, p = .028, η_p² = .063. This interaction was further explored with four separate ANOVAs, one for each emotion category (p values of these follow-ups were adjusted using the Holm–Bonferroni method). Marginally significant two-way interactions were found for elatedness, F(2, 72) = 4.46, p = .06, η_p² = .110, and for surprise, F(2, 72) = 3.70, p = .09, η_p² = .93. Participants experienced an increase in elatedness in the VR condition, but a decrease in the MR and the traditional conditions (0.36, -0.32, and -0.24, respectively), and they experienced an increase in surprise (from pre- to post-test) in the VR and MR conditions, but not in the traditional condition (0.96, 0.76, and 0.20, respectively). The equivalent ANOVA with negative emotions showed no significant 3-way interaction, but there was a significant main effect of Test, F(1, 72) = 11.17, p = .001, η_p² = .134, due to a reduction in negative emotion between the pre- and post-test (1.6 vs 1.4, respectively). There was also a significant main effect of Condition, F(2, 72) = 3.16, p = .48, η_p² = .081, due to overall less negative emotion in the MR than in the VR and traditional condition (1.22 vs. 1.62 and 1.66, respectively).

Table 4.

The Mean DES Ratings for Each Positive and Negative Emotion Category.

	Pre-test			Post-test
Emotion	Trad.	VR	MR	Trad.	VR	MR
Interest	3.32	3.80	4.04	3.40	3.88	4.04
Amusement	2.88	3.04	3.44	2.92	3.24	3.80
Surprise	1.64	2.12	2.24	1.84	3.08	3.00
Elatedness	2.76	2.52	3.32	2.52	2.88	3.00
Sadness	1.88	2.24	1.44	1.60	1.68	1.16
Anger	1.88	1.60	1.20	1.52	1.60	1.20
Fear	1.56	1.56	1.16	1.40	1.56	1.12
Anxiety	2.32	2.08	1.60	1.84	1.88	1.28
Disgust	1.44	1.28	1.12	1.24	1.44	1.20
Contempt	1.56	1.40	1.12	1.32	1.40	1.16

Pre- and post-test scores for simulator sickness were low (overall average: 1.24), and their internal reliability estimates based on Cronbach’s α (see Table 2) were acceptable for measures used in the social sciences (Kline, 2013). A 3 x 2 mixed ANOVA with the factors Condition (traditional, VR, and MR) and Test (pre and post) revealed no significant effects. Hence, simulator sickness did not depend on Condition (1.30, 1.20, and 1.21, respectively), and it did not increase from before to after learning (1.23 vs. 1.24, respectively).

Learning Experience

The internal reliability estimates for the WBLT Evaluation Scale with the three constructs (learning, design, and engagement) were acceptable (Cronbach’s α: 0.87, 0.87 and 0.93, respectively), and similar to the ones reported by Kay (2011). Figure 2 presents the results for the WBLT evaluation scale with the three constructs (learning, design, and engagement), separately for the three learning conditions. Three separate one-way ANOVA’s, one for each construct, revealed a significant main effect of Condition for engagement, F(2, 74) = 4.74, p = .012, η_p² = .116. Post-hoc LSD tests showed that the VR and MR groups reported being significantly more engaged than the traditional group (4.0, 4.1 vs. 3.4, respectively). Similar patterns were found for the design and learning constructs, however, they did not reach significance.

Figure 2.

Mean Rating and SEM (Error Bars) for the Three WBLT Evaluation Scale Constructs, Separately for the Three Learning Conditions.

Technology Evaluation

Technology evaluation questionnaires were only given to participants in the two XR conditions. The Cronbach’s α for the perceived quality scale and its subscales were acceptable (all > 0.91). Table 5 reports the average scores for the three perceived quality dimensions for the VR and the MR applications. There were no significant differences between the two XR conditions. Both systems were generally rated positively on the system usability scale (3.72 in VR and 3.64 in MR, Cronbach’s α 0.90), and on the six UTAUT sub-scales (see Table 5). In the Igroup Presence questionnaire, VR participants reported a significantly higher sense of presence than MR participants (4.34 vs. 3.78, respectively), t(48) = 2.24, p = .030, d = 0.63. The internal reliability for the UTAUT and the Igroup Presence questionnaire was acceptable (Cronbach’s α 0.90 and 0.83, respectively).

Table 5.

Technology Evaluation Questionnaire Means (With Standard Deviations) With Results of Comparisons (t-Test) Between VR and MR Conditions.

Measure	VR	MR	t(48)	p-value
Perceived Quality Scale (5 point)
Ergonomic	3.85 (0.73)	4.03 (0.86)	0.80	.43
Learning	3.51 (0.79)	3.90 (0.93)	1.58	.12
Hedonic	3.88 (0.80)	2.93 (1.00)	0.18	.86
System Usability Scale (5 point)	3.72 (0.78)	3.64 (0.85)	0.35	.73
UTAUT Questionnaire (7 point)
Performance	4.60 (1.51)	4.81 (1.40)	0.51	.61
Effort	5.20 (1.45)	5.40 (1.57)	0.47	.64
Attitude to technology	5.41 (1.35)	5.40 (1.57)	0.54	.59
Anxiety	3.19 (1.36)	5.63 (1.52)	0.78	.44
Self-efficacy	5.08 (1.22)	5.00 (1.67)	0.19	.85
Behavioural Intention	4.35 (2.02)	5.04 (1.79)	1.28	.21
Igroup Presence Questionnaire (7 point)	4.34 (0.88)	3.78 (0.89)	2.24	.03

Discussion

The main aim of the current study was to determine if VR or MR are suitable alternatives to traditional learning methods and if they have any costs or benefits. Significant learning occurred in all three conditions; however, there was no reliable evidence to suggest that VR and MR provide increased learning over traditional methods. As those in the VR and MR conditions performed as well as those in the traditional condition, this does indicate that VR and MR are viable alternatives to traditional learning. This could be beneficial in distance learning situations where traditional learning is not possible. Alternatively, this equipment could be useful as a supplement to traditional learning, to add an active learning component (see Allcoat & von Mühlenen, 2018).

Although the VR and MR participants did not perform better than those in the traditional condition for learning, there was also no evidence of impairment. Participants had little opportunity to familiarize themselves with the equipment, as they were only given verbal instructions on how to use them. In the same length of time that the traditional group had with standard learning materials, the VR and MR groups were able to learn to use new equipment, acclimatize to a simulated 3D environment, and learn as much from the material. As such, learning could potentially be improved with MR and VR than traditional learning if individuals are familiar with the equipment. Alternatively, VR and MR may be well suited as supplementary learning methods, as our results are in line with those from a meta-analysis which found that e-learning-only situations produced an equal amount of learning compared with classroom-only situations, but blended learning (a combination of both) produced better results than classroom-only instruction (Means et al., 2013). Moreover, it has been claimed that during lectures attention tends to wane after approximately 10–15 minutes (Davis, 2009; McKeachie & Svinicki, 2013). In these situations, VR methodologies might be particularly effective in generating increased engagement and improved learning outcomes. Currently, VR technologies are also used to supplement – and not to replace learning technologies, with short VR lessons (3-7 min) that are integrated into the classic lesson flow, in order to make the subject more visual and comprehensive (e.g., MEL Chemistry, see Fahrenkamp-Uppenbrink, 2015).

Our findings also suggest that there can be an emotional benefit of learning in VR, as seen by the DES results with VR scoring higher for surprise and elatedness compared with traditional learning. This supports similar previous findings of improved mood as a result of learning in VR (Allcoat & von Mühlenen, 2018). As student satisfaction is considered to be an important concept (e.g., Elliott & Shin, 2002), this could be considered a considerable benefit of using this type of equipment. Indeed, past research suggests that student satisfaction should be an important outcome for teaching institutions and educators, (e.g., Appleton-Knapp & Krentler, 2006; Thomas & Galambos, 2004), both for the learning benefits and for the idea of students as “consumers”.

VR and MR both performed significantly better than traditional learning on the measure of engagement. Student engagement has also been proposed to be an important factor in student outcomes (Pascarella & Terenzini, 2005; Trowler, 2010), enhanced learning effectiveness (Zhang et al., 2005), and student success outcomes, such as academic achievements and student satisfaction (Kuh, 2001, 2005; Organisation for Economic Co-operation and Development, 2010). Therefore, the observed increased participant engagement in the VR and MR conditions are also benefits of using the equipment in learning contexts.

One potential problem with the use of XR headsets is simulator sickness. Individuals prone to motion sickness or nausea may be wary of using such devices. Our results indicate that neither VR nor MR caused simulator sickness in this context, suggesting that they would be safe to use in-classroom and distance learning. This is likely in part due to the application being well-designed to avoid motion sickness but may also be due to the short sessions with the equipment. Hence, VR and MR may be best suited to shorter, supplementary learning sessions as part of a larger presentation, or for specific activities, as well as distance learning, which can be done at the learner’s own pace.

The results from the Unified Theory of Acceptance and Usage of Technology Questionnaire, from the System Usability Scale, and from the Perceived Quality Scale showed no difference between VR and MR, suggesting that the two systems are equally suitable for this learning context. However, the participants in the VR environment had a higher increase for positive emotions, suggesting that participants enjoyed using the VR more than the MR. VR also produced higher reports of presence than MR, suggesting that being in an enclosed virtual environment leads to higher levels of immersion. These results suggest that VR has a few benefits over and above MR for this type of practical learning.

Future research should consider longer durations, as well as longitudinal impacts of the use of this technology, as in this study the headsets were only used for a short period of time. One hypothesis would be that over time the novelty effect of using the equipment would wear off, and the benefits would decrease. On the other hand, the benefits might increase over time as individuals become more familiar with technology and how to use it. As individuals become more proficient with the equipment, they may find the novelty less distracting and be more able to focus on the learning. Therefore, research considering the long-term effects of the technology is an important future focus. VR and MR may produce better retention than traditional learning methods, as it has been found that constructivist learning increases knowledge retention (Narli, 2011), however, this is a question that needs to be considered in future research.

A further suggestion for future research is the potential to focus more on the individual benefits of MR and VR, rather than taking a directly comparative approach as in this study. The design of both the VR and MR learning environments can be further enhanced to reflect the unique properties of these platforms. For example, MR could be made to interact more directly with the surrounding environment or physical components. VR could more accurately simulate a real workplace environment. These factors, along with others such as audio, were not included in the present study in order to reduce confounding variables, but could be interesting topics for future research.

Conclusions

The impact of this paper includes empirical research conducted into the pedagogic benefits of MR, a tool which has not been rigorously studied for educational applications. As this technology is relatively new and not readily available on the market or easily accessible, little empirical research has been conducted with it. Although MR has some similarities with VR, the ways they are used and applied in educational settings are potentially quite diverse due to the differing natures of the technologies. As well as considering MR alongside VR, a broader range of issues for using these technologies in education were measured. As well as the knowledge-learning aspects, user-technology interaction and user experience were examined to consider if the technology is suitable for classroom use.

The overall results do suggest that VR and MR are both suitable and safe technologies for learning, potentially enabling new approaches to teaching. Applications for these technologies can also be adapted to suit pre-existing courses, both classroom-based and distance learning. Distance learning applications can be particularly beneficial in circumstances such as those during the Covid-19 pandemic. Even when considering possible restrictions of the technology, such as how long it can be used for comfortably, the benefits, such as increased engagement and positive emotions, suggest that VR and MR would be good as supplements to traditional learning methods. As the technology improves, these restrictions will likely be reduced, such as decreased headset weight for improved comfort.

XRs have a myriad of possible uses, as many environments and interactions can be accurately simulated within virtual environments, enabling access to learning materials that may not otherwise be available to learners (Bailenson, 2018). For example, dangerous environments or chemicals can be re-created virtually, or expensive equipment that would be too costly for institutions to purchase. In these cases, XRs can provide learning methods not otherwise available, allowing a more hands-on, immersive, and active learning approach to teaching and learning.

Supplemental Material

sj-pdf-1-jec-10.1177_0735633120985120 - Supplemental material for Education in the Digital Age: Learning Experience in Virtual and Mixed Realities

Supplemental material, sj-pdf-1-jec-10.1177_0735633120985120 for Education in the Digital Age: Learning Experience in Virtual and Mixed Realities by Devon Allcoat, Tim Hatchard, Freeha Azmat, Kim Stansfield, Derrick Watson, Adrian von Mühlenen in Journal of Educational Computing Research

Footnotes

Acknowledgments

We thank Jake Tinsley, Vivien Hung and Emma Prescott for their help with programming and data collection. Thanks also go to Glynn Taylor for programming advice.

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The authors received no financial support for the research, authorship, and/or publication of this article.

ORCID iDs

Devon Allcoat

Adrian von Mühlenen

Notes

1. Computer skills consisted of two combined questions, where participants were asked to rate on six-point scales their confidence about using computers (from “not at all confident” to “completely confident”) and their computer skill level (from “no skills” to “professional skills”). Gaming experience consisted of two combined questions, where participants had to indicate how many hours a week, they were playing video games on average (from “0–1 hours” to “12+ hours”) and to what extent they would consider themselves a “gamer” (from “not at all” to “highly”). Headset experience consisted of one question where they were asked if they ever used a virtual or mixed reality headset before (from “never” to “regularly”).

2. Piloting was undertaken for all three conditions. Pilot tests were completed with students from the engineering course that the learning material was sourced from. Feedback was given to check the learning materials, length of testing time, and navigation of the VR and MR environments.

Supplemental Material

Supplemental material for this article is available online.

Author Biographies

Devon Allcoat completed her PhD on ‘Effects and Applications of Video Games and Virtual Environments’ at the Department of Psychology of the University of Warwick, conducting experimental research on the human factors of the use of video games and virtual environments. She is now a Teaching Fellow at Warwick Manufacturing Group (WMG).

Tim Hatchard is an undergraduate student in Mechanical Engineering at the University of Warwick. His current research interests include virtual reality for training and educational purposes.

Freeha Azmat is a senior teaching fellow and course director for the Digital Technology Solutions Degree apprenticeship at WMG. She is leading several modules related to programming and Electronics in Applied engineering program (AEP) and Dyson engineering degree apprenticeship program.

Kim Stansfield is a principal teaching fellow in Transformational Systems Engineering and leads development and delivery of Warwick University WMG’s Part Time MSc Degree Apprenticeship in ‘Systems Engineering Technical Leadership’. He collaborated with his co-authors in evaluation and optimisation of Technology Enhanced Learning solutions.

Derrick Watson is a professor in the Department of Psychology, University of Warwick. His research interests include the mechanisms of spatial and temporal visual selection and processing, road safety, driver performance and the use of virtual reality and simulation for studying human behaviour and performance.

Adrian von Mühlenen is an associate professor at the Department of Psychology, University of Warwick. He has a general interest in cognitive psychology, including visual perception, attention, learning, memory, and emotion, with a particular interest in how new technologies can be used to advance research in these areas.

References

Akbar

(2013). What affects students’ acceptance and use of technology? Dietrich College Honours Theses. http://repository.cmu.edu/hsshonors/179%5Cnfiles/541/179.html

Albidewi

Tulb

(2014). Promise and challenge of e-learning—Literature review. European Scientific Journal, 10(1), 210–217.

Allcoat

von Mühlenen

(2018). Learning in virtual reality: Effects on performance, emotion and engagement. Research in Learning Technology, 26, 1–13. https://doi.org/10.25304/rlt.v26.2140

Anderson

L. W.

Krathwohl

D. R.

Airasian

P. W.

Cruikshank

K. A.

Mayer

R. E.

Pintrich

P. R.

Raths

Wittrock

M. C.

(2001). A taxonomy for learning, teaching, and assessing: A revision of Bloom's taxonomy of educational objectives. Pearson, Allyn & Bacon.

Andújar

J. M.

Mejías

Márquez

M. A.

(2011). Augmented reality for the improvement of remote laboratories: An augmented remote laboratory. IEEE Transactions on Education, 54(3), 492–500.

Antoniou

P. E.

Dafli

Arfaras

Bamidis

P. D.

(2017). Versatile mixed reality medical educational spaces; requirement analysis from expert users. Personal and Ubiquitous Computing, 21(6), 1015–1024. https://doi.org/10.1007/s00779-017-1074-5

Appleton-Knapp

S. L.

Krentler

K. A.

(2006). Measuring student expectations and their effects on satisfaction: The importance of managing student expectations. Journal of Marketing Education, 28(3), 254–264. https://doi.org/10.1177/0273475306293359

Bailenson

(2018). Experience on demand: What virtual reality Is, how It works, and what It can do. Norton.

Bennie

S. J.

Ranaghan

K. E.

Deeks

Goldsmith

H. E.

O’Connor

M. B.

Mulholland

A. J.

Glowacki

D. R.

(2019). Teaching enzyme catalysis using interactive molecular dynamics in virtual reality. Journal of Chemical Education, 96(11), 2488–2496. https://doi.org/10.26434/chemrxiv.7819982

10.

Benware

C. A.

Deci

E. L.

(1984). Quality of learning with an active versus passive motivational set. American Educational Research Journal, 21(4), 755–765. https://doi.org/10.3102/00028312021004755

11.

Bloom

B. S.

(1956). Taxonomy of educational objectives, handbook I: The cognitive domain. David McKay Co Inc.

12.

Boyle

G. J.

(1984). Reliability and validity of Izard’s differential emotions scale. Personality and Individual Differences, 5(6), 747–750. https://doi.org/10.1016/0191-8869(84)90124-7

13.

Brooke

(1996). SUS—A quick and dirty usability scale. Usability Evaluation in Industry, 189(194), 4–7. https://doi.org/10.1002/hbm.20701

14.

Cheng

D’Angelo

(2018). CIRCL primer: Virtual reality in educational settings. CIRCL Primer Series. http://circlcenter.org/virtual-reality-in-education

15.

Chi

M. T. H.

Wylie

(2014). The ICAP framework: Linking cognitive engagement to active learning outcomes. Educational Psychologist, 49, 219–243.

16.

Comeau

C. P.

Bryan

J. S.

(1961). Headsight television system provides remote surveillance. Electronics, 34, 86–90.

17.

Cuban

(1986). Teachers and machines: The classroom use of technology since 1920. Teachers College Press.

18.

Cui

(2013). An empirical study of learning outcomes based on active versus passive teaching styles. International Journal of Education and Management Engineering, 3(1), 39–43. https://doi.org/10.5815/ijeme.2013.01.07

19.

D’Angelo

C. M.

Rutstein

D. R.

Harris

C. J.

Bernard

Borokhovski

Haertel

(2014). Simulations for STEM learning: Systematic review and meta-analysis. SRI International.

20.

Davies

(2008). Learning and Teaching in Laboratories. Higher Education Academic Engineering Subject Centre.

21.

Davis

B. G.

(2009). Tools for teaching. Jossey-Bass.

22.

Dede

(2009). Immersive interfaces for engagement and learning. Science, 323(5910), 66–69. https://doi.org/10.1126/science.1167311

23.

Dewey

(1938). Experience and education. Macmillan Company.

24.

Duffy

T. M.

Jonassen

D. H.

(2013). Constructivism and the technology of instruction: A conversation. Routledge. https://doi.org/10.4324/9780203461976

25.

Elliott

K. M.

Shin

(2002). Student satisfaction: An alternative approach to assessing this important concept. Journal of Higher Education Policy and Management, 24(2), 197–209. https://doi.org/10.1080/1360080022000013518

26.

Fahrenkamp-Uppenbrink

(2015). A kid’s guide to chemistry. Science, 350(6266), 1322. https://doi.org/10.1126/science.aad9017

27.

Fosnot

(1996). Constructivism: A psychological theory of learning. In C. Fosnot (Ed.), Constructivism: Theory, perspectives, and practice (pp. 8–33). Teachers College Press.

28.

Freeman

Eddy

S. L.

McDonough

Smith

M. K.

Okoroafor

Jordt

Wenderoth

M. P.

(2014). Active learning increases student performance in science, engineering, and mathematics. Proceedings of the National Academy of Sciences, 111(23), 8410–8415. https://doi.org/10.1073/pnas.1319030111

29.

Godden

D. R.

Baddeley

A. D.

(1975). Context-dependent memory in two natural environments: On land and underwater. British Journal of Psychology, 66(3), 325–331. https://doi.org/10.1111/j.2044-8295.1975.tb01468.x

30.

Goertz

R. C.

Mingesz

Potts

Lindberg

(1965). An experimental head-controlled television to provide viewing for a manipulator operator. In Proceedings of the 13th remote systems technology conference (pp. 57–60). American Nuclear Society.

31.

Gumport

P. J.

Chun

(2005). Technology and higher education: Opportunities and challenges for the new era. In P. Altbach, R. O. Berdahl, & P. Gumport (Eds.), American higher education in the twenty-first century (2nd ed., pp. 393–424). Johns Hopkins University Press.

32.

Hilgarth

(2010). Successful e-Learning? A multidisciplinary literature review on e-Learning success. In M. B. Nunes, P. Isaías, & P. Powell (Eds.), Proceedings of the IADIS international conference information systems 2010 (pp. 583–587). IADIS Press. http://www.scopus.com/inward/record.url?eid=2-s2.0-84860777955&partnerID=40& md5=8e1667f85e1faabd1b37ca458c201f68

33.

Izard

C. E.

Dougherty

F. E.

Bloxom

B. M.

Kotsch

N. E.

(1974). The Differential Emotions Scale: A method of measuring the meaning of subjective experience of discrete emotions. Vanderbilt University, Department of Psychology.

34.

Jacobsen

D. M.

(2000). Examining technology adoption patterns by faculty in higher education [Paper presentation]. ACEC2000: Learning Technologies, Teaching and the Future of Schools, Melbourne, Australia. https://people.ucalgary.ca/∼dmjacobs/acec/

35.

Kavanagh

Luxton-Reilly

Wuensche

Plimmer

(2017). A systematic review of virtual reality in education. Themes in Science and Technology Education, 10(2), 85–119.

36.

Kay

(2011). Evaluating learning, design, and engagement in web-based learning tools (WBLTs): The WBLT Evaluation Scale. Computers in Human Behavior, 27(5), 1849–1856.

37.

Kennedy

R. S.

Lane

N. E.

Berbaum

K. S.

Lilienthal

M. G.

(1993). Simulator sickness questionnaire: An enhanced method for quantifying simulator sickness. The International Journal of Aviation Psychology, 3(3), 203–220.

38.

Kline

(2013). Handbook of psychological testing (2nd ed.). Routledge.

39.

Koretsky

M. D.

Magana

A. J.

(2019). Using technology to enhance learning and engagement in engineering. Advances in Engineering Education, 7(2), 1–53.

40.

Kuh

G. D.

(2001). Assessing what really matters to student learning: Inside the national survey of student engagement. Change, 33(3), 10–17. https://doi.org/10.1080/00091380109601795

41.

Kuh

G. D.

(2005). Putting student engagement results to use : Lessons from the field. Assessment Update, 17(1), 12–14.

42.

Markant

D. B.

Gureckis

T. M.

(2014). Is it better to select or to receive? Learning via active and passive hypothesis testing. Journal of Experimental Psychology: General, 143(1), 94.

43.

McKeachie

Svinicki

(2013). McKeachie’s teaching tips (14th ed.). Wadsworth Publishing.

44.

McMahan

(2003). Immersion, engagement, and presence. In Wolf

M. J. P.

Perron

(Eds.), The video game theory reader (pp. 67–86). Routledge.

45.

Means

Toyama

Murphy

Bakia

(2013). The effectiveness of online and blended learning: A meta-analysis of the empirical literature. Teachers College Record, 115, 1–47.

46.

Milgram

Takemura

Utsumi

Kishino

(1995). Augmented reality: A class of displays on the reality-virtuality continuum. In H. Das (Ed.), Telemanipulator and telepresence technologies (Vol. 2351, pp. 282–292). SPIE. https://doi.org/10.1117/12.197321

47.

Mount

N. J.

Chambers

Weaver

Priestnall

(2009). Learner immersion engagement in the 3D virtual world: Principles emerging from the DELVE project. ITALICS: Innovations in Teaching & Learning in Information & Computer Sciences, 8(3), 40–55. https://doi.org/10.11120/ital.2009.08030040

48.

Nadler

R. T.

Rabi

Minda

J. P.

(2010). Better mood and better performance: Learning rule-described categories is enhanced by positive mood. Psychological Science, 21(12), 1770–1776. https://doi.org/10.1177/0956797610387441

49.

Narli

(2011). Is constructivist learning environment really effective on learning and long-term knowledge retention in mathematics? Example of the infinity concept. Educational Research and Reviews, 6, 36–49.

50.

Olmos-Raya

Ferreira-Cavalcanti

Contero

Castellanos

M. C.

Giglioli

I. A. C.

Alcañiz

(2018). Mobile virtual reality as an educational platform: A pilot study on the impact of immersion and positive emotion induction in the learning process. Eurasia Journal of Mathematics, Science and Technology Education, 14(6), 2045–2057. https://doi.org/10.29333/ejmste/85874

51.

Organisation for Economic Co-operation and Development. (2010). PISA 2009 results : Learning to learn-student engagement, strategies and practices (Volume III) (Vol. III, p. 270). https://doi.org/10.1787/9789264083943-en

52.

Pan

Cheok

A. D.

Yang

Zhu

Shi

(2006). Virtual reality and mixed reality for virtual learning environments. Computers and Graphics (Pergamon), 30(1), 20–28. https://doi.org/10.1016/j.cag.2005.10.004

53.

Pascarella

E. T.

Terenzini

P. T.

(2005) How college affects students: A third decade of research (Vol. 2). Jossey-Bass.

54.

Piaget

(1937). La construction du réel chez l'enfant [In English, 1955: The construction of reality in the child]. Delachaux et Niestlé.

55.

Piaget

(1950). The psychology of intelligence. Routledge

56.

Pribeanu

Balog

Iordache

D. D.

(2017). Measuring the perceived quality of an AR-based learning application: A multidimensional model. Interactive Learning Environments, 25(4), 482–495.

57.

Rolfe

Staples

(1986). Flight simulation. Cambridge University Press.

58.

Rosenberg

M. J.

(2001). E-Learning: Strategies for delivering knowledge in the digital age. McGraw-Hill.

59.

Schmid

R. F.

Bernard

R. M.

Borokhovski

Tamim

R. M.

Abrami

P. C.

Surkes

M. A.

Wade

Woods

(2014). The effects of technology use in postsecondary education: A meta-analysis of classroom applications. Computers and Education, 72, 271–291. https://doi.org/10.1016/j.compedu.2013.11.002

60.

Schubert

Friedmann

Regenbrecht

(2001). The experience of presence: Factor analytic insights. Presence: Teleoperators and Virtual Environments, 10(3), 266–281.

61.

Seligman

M. E.

Ernst

R. M.

Gillham

Reivich

Linkins

(2009). Positive education: Positive psychology and classroom interventions. Oxford Review of Education, 35(3), 293–311.

62.

Selwyn

(2017). Education and technology: Key issues and debates (2nd ed.). Bloomsbury Academic.

63.

Thomas

Galambos

(2004). What satisfies students? Mining student-opinion data with regression and decision tree analysis. Research in Higher Education, 45(3), 251–269. https://doi.org/ 10.1023/b:rihe.0000019589.79439.6e

64.

Trowler

(2010). Student engagement literature review. Lancaster University, The Higher Education Academy. http://www.heacademy.ac.uk/assets/documents/studentengagement/StudentEngagementLiteratureReview.

65.

Tukey

J. W.

(1977). Exploratory data analysis. Addison-Wesley.

66.

Valdez

M. T.

Ferreira

C. M.

Martins

M. J. M.

Barbosa

F. P. M.

(2015). 3D virtual reality experiments to promote electrical engineering education. In International conference on information technology based higher education and training (pp. 3–6). Institute of Electrical and Electronics Engineers (IEEE). https://doi.org/10.1109/ITHET.2015.7217957

67.

Venkatesh

Morris

M. G.

Davis

G. B.

Davis

F. D.

(2003). User acceptance of information technology: Toward a unified view. MIS Quarterly, 27(3), 425–478.

68.

Vygotsky

L. S.

(1980). Mind in society: The development of higher psychological processes. Harvard University Press.

69.

Welsh

E. T.

Wanberg

C. R.

Brown

K. G.

Simmering

M. J.

(2003). E-learning: Emerging uses, empirical results and future directions. International Journal of Training and Development, 7(4), 245–258. https://doi.org/10.1046/j.1360-3736.2003.00184.x

70.

Zacharia

Z. C.

Olympiou

(2011). Physical versus virtual manipulative experimentation in physics learning. Learning and Instruction, 21, 317–331.

71.

Zhang

Zhao

J. L.

Zhou

Nunamaker

J. F.

(2004). Can e-learning replace classroom learning? Communications of the ACM, 47(5), 75–79. https://doi.org/10.1145/986213.986216

72.

Zhang

Zhou

Briggs

Nunamaker

J. F.

(2005). Instructional video in e-learning: Assessing the impact of interactive video on learning effectiveness. Information and Management, 43(1), 15–27.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.28 MB