The Transferability of Pilots’ Video Gaming Experience to the Skills and Situation Awareness of Operating UAVs

Abstract

Unmanned aerial vehicles (UAVs), or drones, are increasingly used in various industries. However, this increased automation can lead to reduced safety, decreasing situational awareness (SA). Industry experts have investigated operator accuracy, attentiveness, and selection methods by testing transferable skills between pilots with different video gaming (VG) habits and flying a drone. To contribute to this research further, we investigated pilots’ VG playing habits on SA while flying a UAV and their ability to learn and perform in a novel scenario. SA of non-VG playing pilots and VG playing pilots while operating a drone using a simulator was compared. VG playing pilots (n = 31), compared to non-VG playing pilots (n = 31) were quicker to adapt and learn in a novel environment and exhibited significantly better SA while flying a drone. These findings may be useful to identify individuals who are predisposed to better SA, thus improving selection and training.

Unmanned aerial vehicles (UAVs), or drones, have been implemented in various industries to reduce human involvement in their operations (Custers, 2016). However, increased automation can lead to increased mishaps or reduced safety as it can overload operators with information (Cooke, 2006), leading to reduced situational awareness (SA). As a result, there is a growing interest in human factors in UAV operations, including their accuracy and attentiveness during operation and operator selection methods (Cooke, 2006; McKinley et al., 2011). This interest has prompted industry experts to examine the relationship between playing video games (VGs) and flying drones by testing transferable skills between VG players and pilots to operate a UAV (McKinley et al., 2009, 2011). To contribute to this research further, this study investigated the effect of pilots’ VG playing habits on their SA while flying a UAV. The study also examined the effect of habitual video gaming on pilots’ ability to learn and perform in a novel scenario.

VGs are a ubiquitous part of the lives of millions (Entertainment Software Association [ESA], 2022); therefore, it is important to understand their effect on players’ skills and behaviors. VGs are software supported by display devices that generate audio and visual feedback when users interact through software-specific input devices (Umaschi Bers, 2010). It creates an immersive and emotional experience for the player by allowing them to make decisions that influence the game’s narrative (Oliver et al., 2016), making it a popular entertainment form. About two-thirds of the American population plays VGs for at least an hour per week. Most gamers are in the age range of 18 to 34, with shooter games being their most popular genre (ESA, 2022). There are five genres of commercial VGs: traditional, simulation, strategy, action, and fantasy VGs (Choi et al., 2020), where shooter games fall under the action genre. This study will focus on action video games (AVGs), not only because of their popularity but also because they allow the player complete control over the avatar’s movement and field of view (McKinley et al., 2011), similar to operating a UAV.

AVG and Cognitive Functions

To understand the impact of playing AVGs on SA, it is crucial to understand its effect on the AVG players’ (AVGPs) cognitive functions. Choi et al. (2020) discussed six cognitive functions positively linked to VGs depending on the genre. Out of the six, attention, working memory, visuospatial skills, and problem-solving skills will be discussed as these skills are essential in maintaining SA.

AVGs can improve attentional control in AVGPs. Long-term gameplay enhances AVGPs’ spatial localization and the ability to focus on the target among irrelevant information. Compared to NVGPs, AVGPs have faster processing speeds and can divide their attention to accurately track multiple objects with significantly small or no lapse in attention in a dynamic environment. Although AVGPs were faster than NVGPs, both were equally impulsive, accurate, and attentive (Bavelier et al., 2012).

Working memory (WM) is the process of storing information for short periods while simultaneously completing additional tasks. WM helps to maintain visual stimuli as needed (Baddeley, 2007). Long-term AVG experience improves the visual WM capacity of players (Choi et al., 2020). Compared to NVGPs, AVGPs demonstrated greater short-term visual memory (Sungur & Boduroglu, 2012) and more precise and faster processing of changes in visual information (Boot et al., 2008).

Visuospatial refers to the ability to perceive, recognize, and manipulate visual information. AVGPs showed better navigation skills, visuomotor coordination, and spatial information processing. No significant effect of AVGs was found on problem-solving skills; however, commercial VGs enhanced problem-solving skills in university students (Choi et al., 2020).

SA in UAV Operators

UAV operators must maintain SA to operate the vehicle safely (Drury et al., 2006a). Endsley (1995) defines SA as “the perception of the elements in the environment within a volume of time and space, the comprehension of their meaning, and the projection of their status in the near future” (p. 36). Endsley states that SA occurs at three levels: the first is the operators’ perception of data, the second is the operators’ comprehension of meaning, and the third is the projection of the near future. Most UAV mishaps resulted from human-system integration issues such as a lack of SA. Compared to manned aircraft, UAVs have more mishaps per 1000 flight hours (Tvaryanas et al., 2005). Despite being unmanned, drones require human control and monitoring, which can increase the cognitive workload of operators, leading to a decrease in SA (Ting et al., 2014). Assessing the interactions between the operator and the UAV, including the spatial relationships between the UAV and targets, other aircraft, terrain, and points on Earth (Drury et al., 2006b), may be essential to understanding the cognitive workload and SA of operators.

Gaming and SA in UAV Operators

Cooke (2006) suggests that operating a UAV is similar to a combination of manned aircraft operations and video gaming. VGPs and manned aircraft pilots show different transferrable skills in various aspects of UAV flying (Cuevas & Aguiar, 2017; McKinley et al., 2009, 2011; Wheatcroft et al., 2017). Cuevas and Aguiar (2017) found a significant positive correlation between first-person shooter (FPS) games, which are type of an AVG, and spatial orientation. Wheatcroft et al. (2017) found that VGPs exhibit lower overconfidence in decision judgments when compared to pilots. They also found that pilots and VGPs showed higher decision confidence than the control group. McKinley et al. (2009, 2011) found that pilots perform significantly better than VGPs on multi-attribute cognitive tasks, while VGPs perform significantly better on cognitive tests pertaining to visually acquiring, identifying, and tracking targets. However, both groups performed similarly in the study’s UAS landing task. Both studies concluded that in novel environments, VGPs’ cognitive skills may transfer and improve performance in UAS tasks compared to NVGPs. Since both have different transferrable skills, this current study combined both groups (pilots with VG experience) to understand their cumulative effect on SA while flying a UAV.

The Situation Awareness Global Assessment Technique (SAGAT) is commonly used to measure SA. SAGAT freezes the screen in a simulation and queries participants for information about the scenario (Endsley, 1996). This method allows researchers to evaluate an operator’s situational awareness at three levels. Questions asked using this tool ensure that the answers are unbiased of attention as participants cannot anticipate when the next question will be asked. In addition to accuracy as a measurement of SA, Situation Presence Assessment Method (SPAM) measures response time (RT) as an indicator of SA (Durso & Dattel, 2004). However, the scenario is typically not frozen when SPAM is utilized; therefore, responses are made in real-time. Because RT can indicate processing speed, faster RTs indicate better SA.

It is expected that VGPs should have better SA in a UAV-simulated task when compared to NVGPs. RT (used in SPAM) and accuracy (used in SAGAT and SPAM) can be considered reliable measures of SA. Although SAGAT is a more common measurement in SA research, this study used a hybrid approach to measuring SA. Specifically, SAGAT was employed with the addition of measurement RT to answer SA questions.

Method

Participants

Sixty-five students were recruited at a medium-sized aeronautical university in the southeastern United States. Two participants were removed as they failed to meet the minimum requirements of the study, and one outlier which was greater than three SDs above the mean was removed. The age range of participants was 18 to 35, with the average age being 21.81 year (SD = 3.98). Of the 62 participants, 52 were male and 10 were female. VGPs consisted of 28 males and three females, whereas NVGPs consisted of 24 males and seven females. Participants were recruited via email. All participants received $10 for approximately 20 to 30 min of time commitment.

All participants held a minimum of a private pilot certificate and had no prior UAV experience. NVGPs were defined as less than an average of 1 hr a week of experience playing VGs in the past 6 month. VGPs were defined as participants’ experience playing AVG games for at least 5 hr a week and no >3 hr a week of any other genre for at least 6 month prior.

Materials

At the onset of the experiment, participants were requested to review and sign an informed consent form stating their willingness to participate in this study and complete a demographic questionnaire inquiring about their age, gender, and flight hours, among others. They were then provided with a short demonstration (approximately 10 min) on operating a drone within the drone simulator (DJI flight simulator) using its associated controller (DJI remote controller). This controller allows users to move the drone along its x, y, and z planes and adjust the camera angle of the drone. After the demonstration, the participants were instructed to practice flying the drone for approximately 5 min in the software’s “Time Trial: Racing Route 1” mode to ensure they were comfortable using the controller and operating the drone.

The participants then proceeded with the test scenario, which utilized the software’s “Time Trial: Racing Route 5” mode. Racing Routes 1 and 5 modes resembled a simple timed obstacle course or race where obstacles are avoided by maintaining adequate distance. The aim of the modes was to maneuver the drone through predetermined waypoints (as shown in Figure 1) that directionally guide the drone operator through the route. New waypoints are revealed when the drone is flown through the currently visible waypoint. To successfully complete the test scenario, participants navigated the drone through 19 waypoints (as shown in Figure 1) within 3 min while maintaining a safe altitude and distance from surrounding objects. In this scenario, participants’ performance data (number of collisions and time taken to complete the scenario, if completed) was collected.

Figure 1.

Time Trial: Racing Route 5.

During the scenario, Audacity software was used to ask six SA questions at approximately 30-second intervals. The questions were associated with their progress in the scenario. If participants failed the scenario (crashed or ran out of time) before all six questions were asked, the scenario was restarted. However, any questions that were previously used were not reused once the scenario was restarted. That is, if a participant answered the third SA question, and then crashed, the fourth question would be the first question the participant received after restarting the scenario.

The following are the questions in the order they were asked:

1. Is the next waypoint straight ahead, left, or right?

2. How many waypoints are visible at this point?

3. Is the size of the diamond smaller, bigger or the same size as the waypoint?

4. How many more waypoints are left?

5. How much time do you have left?

6. Does the road go straight, left, or right?

The experiment used the SAGAT method with an element of SPAM to collect the accuracy of responses and RT data to the questions listed above. Each time a question was posed, the screen was frozen to measure their response accuracy, which aligns with SAGAT. However, the researcher also opted to measure RT as an additional measure of SA to ascertain how long participants took to recall the information. Since the participants were unaware that their RT was measured, it did not interfere with the SAGAT method.

Procedure

After signing the consent form and completing a demographic questionnaire, participants were given a demonstration of how to fly the drone in the simulator using the simulator controller. They were then allowed to practice operating the drone for approximately 5 min in the “Time Trial: Racing Route 1” mode. After adequate practice, participants were asked to complete the “Time Trial: Racing Route 5” mode.

Results

Data Preparation and Scoring

Measured variables included SA, number of crashes, and whether the participants could complete all 19 waypoints. Each participant received six SA questions independent of the number of crashes or waypoints completed. Since the scenario was timed for 3 min, SA questions were administered on average every 30 s. Furthermore, each SA question was independent of the participant’s time or position in the scenario. That is, a response to an SA question programed to occur at 1 min in the scenario could just as easily be programed to occur 2 min into the scenario. Because a participant had control of the speed at which they were operating the drone, a correct response for one participant to a question that occurred 1 min in the scenario may have a different correct response to the same SA question administered at the same time for another scenario. Thus, correct responses to SA questions were not tied to a particular time or position in the scenario. A total accuracy score of SA questions was recorded, providing a value range from 0 to 6. The average RT to answer SA questions was also recorded. However, RT to only correctly answered SA questions was included in the data.

The DJI flight simulator is programed to end a scenario if a player crashes the drone. A crash occurs when the drone hits an object on the side of the road (e.g., buildings, trees). A crash also occurs if the participant hits the boundary of a waypoint gate. If a participant crashed the drone, then they started the scenario again. However, any SA question that was asked in a previous scenario was not repeated. The process of restarting the scenario continued until the participant completed all six questions. The number of crashes was recorded, corresponding to the number of scenarios the participant started minus 1 (n – 1). Once the participants answered all six SA questions (despite how many scenarios they had started), they were allowed to continue the scenario to see if they had reached the final waypoint without crashing. Differences between participants who could complete all 19 waypoints (despite the number of scenarios started) and participants who did not complete all 19 waypoints (crashed before completing the scenario after answering the last SA question) were recorded.

No difference in flight time t(56) = 1.293, p = .201 was found between the VGP group (M = 193.80, SD = 205.29) and the NVGP group (M = 307.50, SD = 420.54). Age and flight time showed a positive correlation r(56) = .584, p < .001.

SA

An independent samples t-test t(60) = 3.677, Cohen’s d = .93, p < .001 showed that the VGP group (M = 3.74, SD = 1.06) answered significantly more SA questions correctly (See Figure 2) than the NVGP group (M = 2.84, SD = 0.86). Additionally, after correcting for unequal variances, the VGP group showed better SA t(40.144) = −4.086, Cohen’s d = 1.04, p < .001 (See Figure 3) by answering questions faster (M = 1.12 s, SD = 0.52) than the NVGP group (M = 2.10 s, SD = 1.24).

Figure 2.

Response accuracy to SA questions for group (Error Bars = 2 SE).

Figure 3.

RT to answer SA questions for group (Error Bars = 2 SE).

Times Crashed and Scenario Completion

The number of times participant crashed before answering all 6 six questions did not differ t(60) = 0.256, p = .79 between the VGP group (M = 2.42, SD = 1.29) and NVGP group (M = 2.52, SD = 1.67). However, more individuals from the VGP group (see Figure 4) completed the scenarios compared to the NVGP group Χ²(1) = 4.509, p = .034. None of the 10 females completed the scenario, but 42.3% of the males completed the scenario Χ²(1) = 6.558, p = .01.

Figure 4.

Percent of individuals in each group who completed all waypoints.

Time to Complete Scenario

As expected, the time taken to get through all six SA questions in the scenario and the number of crashes were correlated r(60) = 0.797, p < .001. A correlation between the time to get through all six SA questions and the time to answer each question showed a small correlation, r(60) = .289, p = .023.

Discussion

SA

There was no significant effect of participants’ total flight hours on the time they took to respond to SA questions. VGPs responses to SA-based questions were significantly more accurate as compared to NVGPs. These results suggest that gamers were more attentive to their surroundings as compared to non-gamers while flying a UAV. This is likely due to VGPs’ better spatial information processing (Choi et al., 2020). It can also be attributed to VGPs’ ability to divide their attention to accurately track multiple objects with significantly small or no lapse in attention in a dynamic environment (Bavelier et al., 2012).

It was also found that VGPs responded significantly quicker to SA-based questions as compared to NVGPs. This agrees with existing literature that states VGPs present greater short-term visual memory due to faster processing of changes in visual information (Boot et al., 2008).

The results show that there is a significant difference in SA of VGP and NVGP pilots while flying a UAV. This is likely due to the similarities and transferable skills between playing a video game and flying an aircraft and flying a drone (Cooke, 2006). This finding adds to previous research that found a positive correlation between FPS games and some elements of SA (Cuevas & Aguiar, 2017). This research shows that cognitive skills such as attention, working memory, and visuospatial abilities gained through AVG play are transferable as well as provide VGP pilots an advantage over NVGP pilots due to their better SA and performance.

Times Crashed and Scenario Completion

The number of crashes was also proportional to the time taken to answer all 6 SA questions, which is expected as the scenario would be restarted after each crash. Both groups crashed equally in the simulation, however, VGP group were more likely to complete the scenario. This shows that while both groups crashed, VGPs were quicker to learn and succeed. This may be due to their enhanced spatial localization and faster processing speeds (Bavelier et al., 2012), allowing them to quickly adapt their skills to safely navigating a drone. This suggests that VGPs excel in situations requiring complex cognitive processes, confirming their capacity to navigate and process dynamic environments effectively.

Conclusion

The findings of this study show that VGP pilots exhibit significantly better SA during UAV operations than NVGP pilots. VGPs demonstrated significantly higher accuracy while responding to SA-based questions and lower RT, indicative of their enhanced attentiveness to the surrounding environment during UAV operation. VGP pilots were also quicker to adapt and learn in a novel environment. This is also supported by existing literature emphasizing their short-term visual memory and faster processing of visual information changes. The observed difference also highlights the transferability of cognitive skills acquired through VG play to real-world scenarios.

This study not only highlights the significant difference in SA between VGPs and NVGPs but also emphasizes the practical implications for training and operational contexts. As technology and recreational activities such as gaming continue to intertwine, recognizing and harnessing the cognitive benefits derived from VG play could prove essential in training more proficient UAV pilots, thereby optimizing performance and safety.

Recommendations

Future research can utilize different gaming genres to understand how they may be transferable to UAV operations. Further, different gaming platforms such as computers, consoles, and Virtual Reality (VR) can be compared to understand the efficacy of which method provides better transferability. Future research should also include a more robust experiment to analyze all facets of SA.

Organizations can incorporate training programs that utilize VG mechanics to enhance cognitive skills such as spatial information processing, attention, and working memory. Integrating elements of VG play into UAV training may potentially accelerate skill acquisition and improve SA. Organizations could also consider gaming experience as a positive factor when recruiting UAV operators. This may also help organizations identify individuals who may be predisposed to increased SA.

Supplemental Material

sj-docx-1-pro-10.1177_10711813241260300 – Supplemental material for The Transferability of Pilots’ Video Gaming Experience to the Skills and Situation Awareness of Operating UAVs

Supplemental material, sj-docx-1-pro-10.1177_10711813241260300 for The Transferability of Pilots’ Video Gaming Experience to the Skills and Situation Awareness of Operating UAVs by Rochelle Potdar and Andrew R. Dattel in Proceedings of the Human Factors and Ergonomics Society Annual Meeting