Comparing Virtual Reality Systems: Aspects of Importance for User Experience

Abstract

This paper describes a study aimed at developing and applying a method for comparing the user experience (UX) of virtual reality (VR) systems, in order to identify aspects that differentiate VR systems from a user perspective as well as aspects that users consider important when selecting a VR system. A within-subjects study was conducted, in which twelve participants played a VR game using four different VR systems (i.e., VR headsets and hand controllers). The UX was evaluated through semi-structured interviews and questionnaires. Headset comfort, fit, and perceived image quality as well as unrestricted mobility (i.e., untethered headsets) were identified as important aspects for the UX. Hand controllers that allowed more natural interaction were generally preferred. Image sharpness and headset comfort were considered the most important aspects for selecting a VR system. The study demonstrated that the method can be applied and generate insights into the UX of VR systems.

Introduction

The virtual reality (VR) market is continuously growing and offers an increasing number of high-resolution VR headsets and controllers. VR applications include both casual use for example, for gaming, but also serious use for example, for military and first responder skill training (Hamad & Jia, 2022; Xie et al., 2021). VR headsets and controllers, here referred to as VR systems, may differ with respect to form factor as well as technical properties.

Knowing which aspects of VR systems are important for the user experience (UX) is relevant when selecting a VR system. UX is defined as the totality of the effects felt by a user before, during and after interaction with a system, product, or device and its influence on usability, usefulness, and emotional state (Hartson & Pyla, 2012). Understanding which aspects that are crucial for UX is especially important in serious settings, such as military (Binsch et al., 2021) or first responder training (Binsch et al., 2023). UX-related issues can distract users from the primary training objectives, hindering effective learning and the transfer of skills to real world, high-risk situations, potentially leading to severe consequences.

Previous research has compared VR systems by reviewing technical specifications (Angelov et al., 2020; Mehrfard et al., 2021) and measuring technical properties of VR systems (Mehrfard et al., 2021; Singla et al., 2017). Specific UX aspects, such as usability, comfort, cybersickness, image quality, and controller ergonomics have also been investigated (Bailey et al., 2023; Mehrfard et al., 2021; Singla et al., 2017). However, previous research studied UX aspects separately, not taking into account which of the aspects users consider most important for the UX.

To address this issue, a study was conducted to investigate which aspects of VR systems users consider most important for UX. The aim was to develop and apply a structured methodology for comparing the UX of VR systems. The study had two research questions, (1) “Which aspects of UX led participants to prefer one VR system over another?” and (2) “Which aspects of VR systems were considered most important when selecting a VR system?”

Background

Technical properties of VR systems have been shown or suggested to influence UX; for example, display properties, such as field of view, resolution, and refresh rate (Souchet et al., 2023; Vlahovic et al., 2022); tracking and whether the headset is tethered or standalone (Vlahovic et al., 2022); and ergonomic aspects for example, weight and form factor (Mehrfard et al., 2021; Souchet et al., 2023; Vlahovic et al., 2022).

Relationships between technical properties of VR systems and specific experiences have been suggested, for example, between display quality, sense of immersion, and sense of presence (Mehrfard et al., 2021; Vlahovic et al., 2022). Further, research has proposed that a high degree of specific experiences, for example, sense of immersion and sense of presence relates to a positive UX (Kim et al., 2020; Suh & Prophet, 2018). Specific experiences related to a negative UX, such as cybersickness and fatigue, have also been identified (Mehrfard et al., 2021; Souchet et al., 2023; Vlahovic et al., 2022).

VR has been used for skill training in high-risk professions—such as military operations and skill training for first responders—to provide safe, controllable, and cost-effective training environments (Abbas et al., 2023; Xie et al., 2021). Research indicates that VR-based training generally leads to successful learning outcomes and effective transfer of skills (Abbas et al., 2023; Xie et al., 2021). However, concerns have been raised regarding the potential for negative transfer in poorly designed training programs, for example, if the VR training tasks does not sufficiently resemble real-world tasks (Xie et al., 2021). Further, understanding and addressing individual experiences during the application of VR training have been argued to be crucial for optimizing the transfer-of-training (Xie et al., 2021).

Methods

To compare different VR systems, the study had participants engage in the first person shooter (FPS) game Half-Life: Alyx. The study used a within-subjects design and a Latin square method was applied to balance the order of the conditions (i.e., the different VR systems).

Four VR systems were used: Meta Quest 3, Valve Index, and HTC Vive XR Elite with corresponding hand controllers, and a Varjo Aero headset with Valve Index controllers since the Aero headset has no corresponding controllers (see Table 1 for technical specifications). The VR systems’ built in speakers or headphones were used when possible, and the Aero headset was used with KOSS Porta Pro external headphones. The participants were not informed about the VR systems’ manufacturer or retail price during the study. The study utilized a VR play area of roughly 3 × 2.5 m.

Table 1.

Technical Specifications of the VR Systems Included in the Study.

Property	Aero	Index	Quest 3	XR Elite
Headset
Resolution	2880 × 2720	1440 × 1660	2064 × 2208	1920 × 1920
Refresh rate	90 Hz	120 Hz	90 Hz	90 Hz
Field of view	115°	108°	110°	110°
Audio	External headphones	Integrated headphones	Integrated speakers	Integrated speakers
Weight	746 g excl. headphones	868 g	520 g	602 g
Connection	Tethered	Tethered	Standalone	Standalone
Hand controllers
Finger tracking		Yes	No	No
Strapped to hand		Yes	No	No

Two computers with identical specifications (Intel Core i7 processor, 32 GB RAM, SSD hard drive, NVIDIA GeForce RTX 2080 Ti Graphics card) were used to run the game. The standalone headsets were connected to the computers wirelessly over Wi-Fi 6. Both the computer and Wi-Fi specifications met or exceeded the recommended requirements for running both the game and VR systems.

Participants

Twelve participants (4 female, 8 male, mean age = 27 years) participated in the study. Eleven participants had previously used VR and ten had played FPS games, at most about once a month. The participants gave informed consent before participating in the study.

Materials

The materials included a pre-study questionnaire, an in-study interview and questionnaire conducted for each condition, and a post-study interview and questionnaire. The pre-study questionnaire concerned demographics and previous experience of VR and gaming. The in-study interview was semi-structured with questions about the participants’ general experience of both headset and controllers. The in-study questionnaire contained questionnaire items regarding sense of presence, tracking ability, fit and comfort for both headset and controllers, visual quality, and audio quality (Table 3). Some questionnaire items were adapted or inspired from the iGroup Presence Questionnaire (iGroup, n.d.) and the Presence Questionnaire (Witmer & Singer, 1998). All items were rated on a 1 to 7 Likert scale (1 being least and 7 most favorable). Since the Index controllers were used with both the Aero and Index headset, the items regarding controllers were only included the first time Index controllers were used. In the semi-structured post-study interview, participants selected their most and least preferred VR headsets and controllers and explained the reasons for their selections. Additionally, the interview examined which aspects the participant considered most important for selecting both a headset and controllers. In the post-study questionnaire, the participants were instructed to rate the headsets and controllers from best to worst. The participants were also asked to select the five aspects they considered most important when selecting a VR system. Pre-defined aspects were provided with the option to add aspects; however, no participant added any additional options (Table 4). The aspects were not ranked.

Procedure

The participants first completed the pre-study questionnaire. The study started with a short practice scenario (3–5 min). The participants were instructed to adjust headset interpupillary distance for each headset prior to use. Thereafter, the participants performed four runs in the test scenario (5–7 min each), using each of the four VR systems. The test scenario included elements relevant to evaluate the VR systems, such as moving and looking around in the virtual environment, picking up and interacting with objects, and moving in real life (e.g., by crouching). A test leader assisted the participants if they struggled to proceed in the scenario, and urged the participants to slow down and explore the virtual environment if they rushed through the scenario. Teleportation was used for locomotion. After each run, the in-study interview and questionnaire were conducted. After all four runs were completed, the post-study interview and questionnaire were conducted.

Analysis

The interviews were recorded and transcribed, initially using Whisper followed by manual reviews by the authors. The resulting transcripts were then analyzed by thematic analysis to identify recurring insights across participants. Two of the authors coded all interviews independently and then identified themes together based on the codes.

Jamovi version 2.3.28 was used for the statistical analysis. The Likert scale ratings in the questionnaires were analyzed using the Friedman test and Durbin-Conover pairwise comparisons (non-corrected) for post hoc analysis. One participant’s ratings for items concerning comfort for the Aero headset were excluded from the analysis because the headset had been improperly fitted.

Results

Several UX aspects were identified as key differentiators between the VR systems, mainly through the interviews. For headsets, comfort, fit, and perceived image quality were the aspects perceived as most important for creating a positive UX, but the increased mobility offered by the standalone headsets was also highly valued. Participants preferred controllers that offered natural interaction, that is, interfaces that feel intuitive and effortless by closely mirroring real-world actions. Table 2 presents the participants’ ratings of the headsets and controllers.

Table 2.

Rating Frequency for the Headsets and Controllers, Where First Is Best.

Rating	Aero	Quest 3	Index	XR Elite
Headset
First	6	4	1	1
Second	2	3	3	4
Third	2	2	7	1
Fourth	2	3	1	6
Hand controllers
First		3	8	1
Second		5	2	5
Third		4	2	6

Headset comfort and fit were considered important for the UX according to the interviews. Aero and Index, both large headsets with substantial padding, were generally perceived as comfortable by the users. In contrast, XR Elite was considered uncomfortable due to the need for a tight fit against the face to maintain stability. Some participants noted that headsets with an overhead strap or similar support over the top of the head, which the XR Elite lacked, improved stability and comfort. Additionally, the participants preferred screw knobs over hook-and-loop fasteners for securing the headset, as screw knobs facilitated easier fit adjustments.

Interview responses also indicated that perceived image quality was an important factor influencing UX. Aero and Quest 3, primarily Aero, had the highest perceived image quality, followed by XR Elite and Index. However, the perceived difference between headsets was not clear-cut and the participants’ opinions varied, where some even considered Index to have the best image quality.

Participants favored standalone headsets because they offered enhanced mobility and were perceived as lighter. However, this aspect was brought up less often in the interviews compared to the comfort and image quality aspects. Some participants expressed that the cable at times interfered with their mobility, thus breaking their immersion.

During the interviews, participants expressed divided opinions regarding their preferred audio setup. Some participants preferred external headphones for their superior audio quality, while others favored integrated audio because external headphones were seen as cumbersome by adding an extra component to the headset.

Regarding controllers, many participants expressed a preference for Index in the interviews since they were considered to offer more natural interaction, as objects could be dropped in-game by letting go of the controllers (since a strap over the back of the hand still kept the controllers in place). In addition, the Index controllers enabled individual finger tracking through touchpads. However, the Index controllers were noted to have a learning curve for mastering these features. Some participants also disliked the reduced tactile feedback from the touchpads compared to buttons.

All controllers were generally considered comfortable, although participants’ preferences varied. Some considered the Quest 3 controllers too small while others considered the XR Elite controllers too big. The XR Elite controllers were often regarded as somewhat cumbersome, primarily because of the ring positioned at the top of the device.

Mean ratings and standard deviations for all in-study questionnaire items are presented in Table 3. Statistically significant differences were identified for two items: item 9 regarding comfort, χ²(3) = 10.742, p = .013, with Aero rated significantly higher than both Quest 3 (p = .009) and XR Elite (p = .001); and item 11 regarding light exclusion, χ²(3) = 13.146, p = .004, with Aero and Index rated significantly higher than Quest 3 (p < .001 and p = .004 respectively), and lastly Aero rated higher than XR Elite (p = .011).

Table 3.

Mean Ratings and Standard Deviations for All In-Study Questionnaire Items.

Item	Aero	Index	Quest 3	XR Elite
Headset
1. I had a sense of “being there” in the virtual world	5.83 (0.58)	5.33 (1.07)	5.83 (1.12)	5.50 (1.00)
2. I was unaware of my physical surroundings when I was in the virtual environment	4.75 (1.60)	4.75 (1.42)	5.33 (1.30)	5.42 (1.00)
3. The virtual world seemed real	5.58 (0.79)	5.25 (1.06)	5.50 (1.45)	5.00 (1.13)
4. Items moved in a natural way (continuous smooth movement)	6.08 (0.52)	5.83 (0.84)	5.67 (1.44)	5.42 (1.00)
5. I could visually distinguish details on items without problems	6.42 (0.79)	5.50 (1.00)	6.00 (0.95)	5.58 (1.51)
6. I could visually distinguish items from the background without problems	6.42 (0.67)	5.83 (0.94)	6.00 (0.85)	5.83 (1.12)
7. The views movement in the virtual environment mirrored the actual movements of my head	6.33 (0.65)	6.25 (0.62)	6.42 (0.90)	6.17 (0.84)
8. The views movement in the virtual environment mirrored the actual movements of my head without delay	6.33 (0.65)	5.92 (0.67)	6.25 (0.75)	5.92 (0.90)
9. The headset sat comfortably on my head*	6.18 a (0.75)	5.67 (1.23)	5.25 (1.22)	4.58 (1.56)
10. I was not troubled by the headsets form	5.55 a (1.37)	5.83 (1.12)	5.17 (1.19)	4.92 (1.08)
11. The headset completely blocked out light from the environment**	6.27 a (1.19)	5.25 (1.71)	3.92 (1.62)	4.92 (1.98)
12. I could identify sounds in the virtual environment without problems	6.00 (1.04)	5.83 (0.84)	5.58 (1.44)	5.17 (1.40)
13. I could localize sounds in the virtual environment without problems	5.50 (1.57)	5.33 (1.16)	5.00 (1.48)	5.08 (1.44)
Hand controllers
14. With the hand controllers, I could perform all actions needed (e.g., use tools and weapon, lift objects) without problems		4.92 (1.17)	5.42 (1.24)	5.50 (0.80)
15. The hand controllers buttons and controls were placed suitably		5.33 (1.16)	5.75 (1.42)	6.00 (0.74)
16. The hand controllers buttons and controls were designed suitably		4.92 (1.83)	5.75 (1.14)	5.92 (0.79)
17. The hand controllers were comfortable to hold		6.00 (0.74)	5.42 (1.51)	5.92 (0.79)
18. I could move (movable) objects without problems with the hand controllers		5.33 (1.07)	5.83 (1.38)	5.92 (1.00)
19. I could manipulate (manipulable) objects without problems with the hand controllers		5.33 (1.07)	5.83 (1.19)	5.75 (0.75)
20. The hands movement in the virtual environment mirrored my actual hand movements		5.75 (1.22)	5.50 (1.24)	5.50 (1.31)
21. The hands movement in the virtual environment mirrored my actual hand movements without delay		5.92 (0.90)	5.67 (1.16)	5.92 (1.00)

a

One participant’s rating was excluded.

*

p < .05. **p < .01.

The most important aspect when selecting a VR system according to the participants was image sharpness, closely followed by headset comfort (Table 4). Headset comfort was considered particularly important for extended use according to the interviews. The participants also considered a secure fit of the headset, freedom of movement, natural interaction with virtual objects and well-placed controller buttons important when selecting a VR system. The results from the post-study interview were in line with the post-study questionnaire.

Table 4.

Frequency of Selected Aspects in the Post-Study Questionnaire.

Aspect	Frequency
That the image is sharp	11
That the headset is comfortable	9
That the headset stays in place	6
That I can move around freely in the physical room	6
That interaction with objects by the hands is natural	5
That buttons and controls on the hand controller are well placed	5
That motion tracking is exact	4
That motion tracking is instant	4
That visual objects move naturally (continuous, smooth movement)	3
That the hand controller is comfortable to hold	3
That the sound quality is good	2
That the field of view is broad	1
That the headset excludes ambient sound	1

Discussion

The study developed and applied a methodology to compare VR systems in terms of UX. This approach enabled identification of both distinguishing aspects among the tested VR systems, and key aspects considered important by users for selecting a VR system.

The study provides a first step in the development of a method for testing VR systems, for example, regarding their applicability for skill training in high-risk professions. The test battery considers a number of human-centric requirements regarding user interaction and comfort. Considering such requirements may allow the users to fully focus on training and not be distracted by UX issues, thus potentially improving transfer-of-training and learning in VR.

In relation to research question 1 regarding UX aspects influencing the preference for one VR system over another, comfort and fit emerged as the main aspects that distinguished the VR headsets, both in the interviews and questionnaires. The larger headsets with generous padding were generally perceived as more comfortable, suggesting that providing a soft and cushioned experience is important for UX. It is important to note that the study scenario was relatively short. In scenarios involving prolonged use, poor comfort may be an even more severe issue but other factors such as weight may also have a greater impact on the overall UX.

While the participants regarded perceived image quality as an important factor, it did not distinguish between the headsets as clearly as comfort and fit. While headsets with higher resolution were generally favored in the interviews this preference was not consistently reflected in the interviews, and did not reach statistical significance in the questionnaire results. This implies that users may not clearly notice differences in resolution such as between the headsets in the study, or that other factors may affect the perceived image quality. One such factor may be fit, since poor fit may cause the headset to get dislocated, thereby decreasing perceived visual quality.

Although the participants preferred headsets to be standalone for increased mobility, the participants rated the Aero headset highest which suggests that they accepted a trade-off of getting better comfort and perceived visual quality, but with a tethered headset. However, the importance of using standalone headsets may differ in different settings, for example, if considerable physical movement is required, a tethered headset might be limiting and thereby influence UX more negatively.

Providing natural interaction was considered important for the hand controllers in the interviews. Features such as being able to release the controller to drop objects in game, by having the controller strapped over the back of the hand, and individual finger tracking through touchpads seemed to enable more natural interaction. However, these features were considered to take some time to get used to, compared to controllers with push-buttons. Thus, the trade-off between having more natural interaction and getting used to such controllers may lead to practical considerations of whether time spent for familiarization with the controllers are worthwhile. Noteworthy, the rating of the Index controllers may have been increased since they were used twice in the study, especially when considering that they were regarded as having a learning curve. Hand sizes may have influenced how participants assessed controller comfort. However, no such data was collected.

In relation to research question 2 regarding which aspects were considered the most important when selecting a VR system, aspects align with the aspects addressed in research question 1. Image sharpness was considered the most important aspect. One consideration is determining the level of technical specifications required to deliver a sufficiently sharp image. That the participants could not fully differentiate visual quality of the headsets in the study even when technical specifications differed suggest that better technical specifications does not necessarily mean better perceived visual quality. Thus, headset testing may be needed to determine if the image is perceived as sharp, rather than just reviewing technical specifications. Headset comfort and fit were also considered important in selecting a headset, which is also headset qualities that may not be apparent in technical specifications, further strengthening the need for testing when selecting VR systems.

Though the study demonstrates that the developed method can be applied for the evaluation of UX satisfaction, it may need to be adapted for specific use cases. In this study, the method was applied to acquire general insights, but adaptions may be required to acquire specific insights for a use case.

Utilizing an appropriate scenario is crucial for effectively comparing and testing VR systems. The scenario used in this study was designed to be a generic VR first-person gaming scenario, including many different elements relevant for evaluating VR systems to enable identification of differences related to those elements. However, the scenario in this study was rather short (5–7 min), and for use cases that require VR use for longer periods the scenario length should be increased to identify aspects that are deemed crucial for the UX in such prolonged use cases. Further, additional elements may be important to include for other use cases, such as carrying out specific tasks or performing certain skills.

Since this study’s results are based on comparisons of four specific VR systems, the findings may not be directly generalizable. Future comparisons including other VR systems are needed to confirm these findings and possibly identify other aspects that are crucial for the UX. A further issue for the generalizability of the current findings is the risk for (reverse) halo effects, that is, that the overall experience of using a VR system may affect whether specific aspects of that system are considered favorable or not. Thus, the specific findings for different aspects of the UX may have been influenced by the overall experiences of the systems used. Generalizability may be higher for research question 2 regarding aspects the participants considered most important, since they are not directly connected to the VR systems used.

For further research, the method’s sensitivity, reliability, and validity requires further attention. The relationship between various UX aspects and user performance, transfer-of-training, and learning in VR scenarios relevant to high-risk professions should also be explored further. Moreover, the relative impact of specific technical aspects on the sense of immersion and presence in VR warrants further investigation.

Conclusions

This study provides a first attempt to develop a method for comparing VR systems grounded in different aspects of UX. The study demonstrated that this method could be used to identify prominent UX aspects. Headset comfort, fit, and good perceived image quality emerged as important aspects, both for providing a good UX and when selecting a VR system. Unrestricted mobility (by using an untethered headset) and use of controllers that allows more natural interaction were identified as the second most important aspects, both for the UX and selection of a VR system. Although further validation of the method is needed, the method shows potential for identifying important UX aspects for VR systems as well as for helping to select VR systems in different use cases, for example, in VR training for high-risk professions.