Evaluating Head-Mounted Visual-Haptic Displays for Recovery from Unusual Flight Attitudes under Normal and Visually-Degraded Conditions

Abstract

Haptics are a promising approach for improving situation awareness when visual and auditory channels are saturated. This study explores using head-mounted haptic feedback to improve pilot recovery from unusual attitudes. Seven participants were tested in Digital Combat Simulator (DCS) under two test conditions (haptic-visual, visual-only) in two environmental conditions (daytime, nighttime).

Introduction

Background

Maintaining situation awareness is essential for safety in piloted aircraft; however in crowded visual environments it can be challenging to attend to and perceive the relevant information (Sarter, 2000). One approach, following the multiple resource model (Wickens, 2008), is to expand information to other sensory resources. In modern-day cockpits, this primarily has been enacted with auditory cues. The auditory channel is well-suited for alarms because auditory alerts do not require active attention, unlike visual alerts. Some examples of auditory alarms include altitude alerter tones and ground-proximity warning systems (GPWS), which provide aural instructions like “TERRAIN. PULL UP.” The prevalence and variety of auditory alerts combined with radio communications and crew coordination can render the auditory channel overloaded.

One approach to addressing auditory or visual saturation is to expand into the haptic domain. Haptic displays have been found to be particularly useful when “visual or audio information is unavailable or deteriorated” or “the user’s sensory capacity is overloaded” (Choi & Kuchenbecker, 2013). There are few active haptic interfaces in most cockpits aside from the stick shaker, which vibrates to warn of impending stall. Prior work on using haptics for situation awareness in aviation has proven reasonably successful. In 2000, haptic cues were demonstrated to improve spatial awareness while hovering a helicopter (Raj et al., 2000).Shortly after, the US military began testing a haptic device called TSAS (Tactile Situation Awareness System), a vibrotactile belt that provides pilots with an understanding of their surroundings haptically. The TSAS device was shown to assist helicopter pilots landing in visually-degraded states (Jennings et al., 2004). Helicopter pilots using an updated version of TSAS were able to “non-visually hover helicopters,” and the technology improved situation awareness and decreased pilot cognitive load (Rupert et al., 2016).

Purpose

The purpose of our experiment was to answer the following research question: “Can additional haptic input improve pilot recovery from unusual flight attitudes?” The FAA’s Airplane Flying Handbook defines an unusual attitude as pitch beyond +25° or −10° or bank angles greater than 45°. We selected the task of recovery from unusual flight attitudes because the maneuvers used to recover are not part of normal flight operations and likely to atrophy over time. We hypothesized that recovery time would be shorter with the addition of haptic feedback since a haptic alarm can directly indicate the action needed without requiring the time to rebuild situation awareness. A meta-analysis by Prewett et al. (2012) found that users benefited from haptic-visual inputs compared to visual-only. Additionally, we hypothesized the effect of haptics would be more pronounced in low-visibility environments where spatial disorientation and loss of situation awareness are more common.

A secondary goal was to investigate the efficacy and comfort of head-mounted haptics in the context of aviation. Previous research has demonstrated the efficacy of head-worn haptics under visually-degraded conditions (Berning et al., 2015; Bertram et al., 2013); however, there has not been extensive work on head-mounted haptics as a part of a multisensory display. Pilots routinely wear headwear such as helmets and headsets, so a head-mounted haptic system potentially has a lower barrier of adoption compared to a new type of body-mounted wearable device.

Methods

We performed an exploratory study with human participants to investigate our research questions. The task was to correct from an unusual flight attitude to straight and level flight as quickly as possible. We tested seven participants meeting the criteria of at least 5 hours of flight experience to mitigate learning effects during task execution.

Setup & Hardware

Participants were tested using Digital Combat Simulator (DCS) simulating a Yak-52. Participants viewed the simulation on a 27″ Dell monitor and used a Microsoft SideWinder Force Feedback 2 Joystick, Virpil APC ACE Flight Pedals, and Virpil VPC MongoosT-50CM3 Throttle for flight control (Figure 1).

Figure 1.

Simulator flight controls.

Haptic cues were provided through a prototype bone-conduction haptic device installed in a baseball cap (Figure 2). When activated, the haptic device simulated tapping on the wearer’s head in any of four locations: anterior, posterior, left temporal, or right temporal. Participants were instructed that maneuvering “away” from the tapping would achieve straight and level flight (e.g., tapping on the forehead would indicate pulling the stick “aft”) following (Brill et al., 2014). Tapping was activated when pitch exceeded +25°/−10° or roll exceeded 45°. When both roll and pitch thresholds were exceeded, the system prioritized zeroing roll before correcting pitch, to avoid overstressing the plane.

Figure 2.

Prototype bone conduction haptic device.

Procedure

Before the task, participants flew freely for 6 min to familiarize themselves with Yak-52 flight characteristics and the haptic cues. Two levels of the test condition (haptic-visual, visual-only) were tested at two environmental conditions (daytime, nighttime), yielding four total test conditions. For each test condition, three unusual flight attitudes were tested for a total of twelve trials per participant (Table 1). The order of conditions were counterbalanced among participants. Following the task, users completed a NASA TLX and qualitative preference survey.

Table 1.

Summary of Conditions Tested.

Condition	Daytime	Nighttime
Haptic feedback	• Pitch up, bank left/right • Pitch down, left/right Inverted, nose low	• Pitch up, bank left/right • Pitch down, left/right Inverted, nose low
No haptic feedback	• Pitch up, bank left/right • Pitch down, left/right Inverted, nose low	• Pitch up, bank left/right • Pitch down, left/right Inverted, nose low

Results

Time to Recovery

We hypothesized that users would recover more quickly in the haptic-visual condition compared to the visual-only condition. In order to calculate this, we subtracted the visual-only by the haptic-visual to calculate the difference in recovery time (μvisual-only -μvisual-haptic). We found no statistically significant difference in recovery times between the visual-only and haptic-visual conditions using a Friedman Test (μ = −0.36, σ = 1.44, p = 0.892).

We observed inter-individual differences in response to the haptic feedback (Table 2). Anecdotal evidence from participants suggests lack of improvement in the haptic-visual condition due to participants’ perception of the indication direction (moving away from the haptic input) as unintuitive. One participant stated that he accidentally moved toward the tapping multiple times.

Table 2.

Inter-Individual Differences in Response Time.

Participant	1	2	3	4	5	6	7
Mean difference (seconds)	2.77	0.16	−3.10	1.42	−2.37	−0.81	−3.09

One notable result was that the standard deviation of time to recovery was much larger for the night condition (σ-day = 0.63, σ-night = 2.24) (Figure 3). This may support our hypothesis that haptic inputs have a greater impact on recovery time in visually-degraded conditions. During the day, pilots performed similarly. However, at night the magnitude of the effect of adding haptics was much larger.

Figure 3.

Large standard deviation difference (μvisual-only- μhaptic-visual) of time-to-recovery.

NASA TLX and User Preference Survey

We calculate the results of the NASA TLX survey by subtracting visual-only by the haptic-visual to calculate the difference in rating. We found that on average, participants had higher workload when haptics were added haptics, but this result was not statistically significant (μ = −5.75, σ = 5.52, p = 0.171). This may be due to the unintuitive indication direction discussed previously.

For our user preference survey (Table 3), we found that users had a slight preference for using the device (μ = 4.57 on a 7-point Likert-scale, σ = 1.72). Notably, multiple participants found the device annoying during the day but helpful at night when visual cues were ambiguous. Since we asked these questions at the end of the study, the responses represent a combination of both day and night conditions.

Table 3.

Summary of User Preference Survey.

Statement	Mean	Std. Dev.
I would wear this device again.	4.00	1.29
The information I was receiving from the device was clear.	4.43	1.27
The sensations produced by this device were distracting.	3.71	1.38
My performance with the device active was better than my performance when it was inactive.	4.57	1.72

Recommendations for Future Work

This study illustrates the importance of a user’s mental representation and interpretation of the meaning of the haptic information. With sufficient training, muscle memory can be built; however, providing users with an appropriate mental representation can significantly reduce the need for flight training by making stimuli intuitive. This mental representation also includes the selection of the proper corrective action and ability to project the results of that action into the future vehicle state. During the exercise with unusual pitch and roll, the pilot who had trained on fighter jets commented “[the haptics] seemed to indicate the wrong direction” since he was taught to perform the maneuver as one action instead of correcting roll and then pitch. Further study comparing the effects of haptics in novice versus experienced users could facilitate a more holistic design-plus-training approach.

One way we attempted to make the mental model more intuitive was to make the haptic input sharper by adjusting aspects of the waveform. One participant who performed well described imagining the haptic stimulus as a bee sting and wanting to move away from it as a result. Future studies should investigate how different forms of haptic feedback (e.g., square wave vs. sinusoidal inputs) influence the speed and accuracy of comprehending the signal.

One additional finding was the significance of context when providing haptic cues—users found haptics annoying during the day but helpful at night. Instead of having haptics that constantly transmit information, we recommend having haptics conditioned on sensory context (i.e., only providing haptic information when the visual field is ambiguous).