Do Head-Mounted Augmented Reality Devices Affect Muscle Activity and Eye Strain of Utility Workers Who Do Procedural Work? Studies of Operators and Manhole Workers

Abstract

Objective

The objective was to determine the effect of two head-mounted display (HMD) augmented reality (AR) devices on muscle activity and eye strain of electric utility workers. The AR devices were the Microsoft HoloLens and RealWear HMT-1.

Background

The HoloLens is an optical see-through device. The HMT-1 has a small display that is mounted to the side of one eye of the user.

Method

Twelve power plant operators and 13 manhole workers conducted their normal procedural tasks on-site in three conditions: HoloLens, HMT-1, and “No AR” (regular method). Duration of test trials ranged up to 30 s for operators and up to 10 min for manhole workers. Mean and peak values of surface electromyographic (sEMG) signals from eight neck muscles were measured. A small eye camera measured blink rate of the right eye.

Results

In general, there were no differences in sEMG activity between the AR and “No AR” conditions for both groups of workers. For the manhole workers, the HoloLens blink rate was 8 to 11 blinks per min lower than the HMT-1 in two tasks and 6.5 fewer than “No AR” in one task. Subjective assessment of the two AR devices did not vary in general.

Conclusion

The decrease in blink rate with the HoloLens may expose utility manhole workers to risk of eye strain or dry-eye syndrome.

Application

HMD AR devices should be tested thoroughly with respect to risk of eye strain before deployment by manhole workers for long-duration procedural work.

Keywords

augmented reality utility muscle activity eye strain eye fatigue blink rate

Introduction

Augmented reality (AR) is a technology that presents virtual information of a task or environment to a user while they see a direct view of the real world. AR is different than virtual reality (VR) because with VR a user sees a totally artificially created environment, such as walking on the moon, while with AR a user sees the real world directly with additional information displayed virtually in the field of view (“augmented” informational channels).

A common type of AR technology is an optical see-through (OST) head-mounted display (HMD), in which the user’s direct view of the world is augmented with the projections of computer-generated content on a semi-transparent display within the user’s visual field (Condino et al., 2020). Common AR HMDs are the Microsoft HoloLens (an OST) and RealWear HMT-1, which has a small display mounted in front of a user’s eye.

AR devices have been shown to improve productivity for procedural tasks when compared to using paper instructions. Braly et al. (2019) found that the HoloLens decreased task completion times for a simple wiring task on a space station. Other studies showed that AR devices increase productivity for more complex tasks, such as toy assembly (Tang et al., 2003), assembling a motor combustion chamber (Henderson & Feiner, 2011), and assembling a car door (Wiedenmaier et al., 2003).

In the space station study (Braly et al., 2019), the participants reported that the HoloLens was heavy, presented a limited field of view, and caused slight eye strain. To the authors’ knowledge, no studies have been conducted that assessed the effect of wearing the HoloLens or other HMD AR devices on muscle fatigue and eye strain (EPRI, 2018). These two occupational health issues need to be addressed before deployment of AR devices to workers, particularly field workers who are at risk of acute trauma, such as a fall or electrical shock. In addition, the long-term effects of AR usage on employees who use the devices for sustained periods of time during a work shift is not known.

The present study tested how the HoloLens and HMT-1 affected muscle activity and eye strain of two groups of electric utility field workers: power plant operators and manhole workers. These two groups of workers were selected for testing because they are likely to be initial users of HMD AR devices in the electric utility sector (if the devices were approved). Plant operators and manhole workers perform procedural tasks that are both simple and complex, and they rely on paper instruction sheets for the more complicated tasks. AR technology can display this type of detailed procedural information in the user’s visual field, which would eliminate the use of paper and minimize memorization of task details.

AR has the potential to grow rapidly in the industrial field work sector and may be used by a substantial percentage of workers in the next 20 years (Porter & Heppelmann, 2017). A deep understanding of how AR systems affect occupational health and safety issues is imperative before AR is used by field workers, particularly industrial and utility workers.

Literature Review

Although field workers, such as construction, maintenance and utility, are likely to be initial users of AR technology, no published studies were found that tested how AR systems affect muscle fatigue and eye strain (EPRI, 2018). These types of workers are likely candidates for AR deployment because their work tends to be procedural and because productivity improvements with AR may have a short payback period due to the workers’ relative high pay. Before AR is deployed to these workers, it is important to know whether the extra weight of AR HMD devices increase neck muscle activity and whether the more focused visual concentration required by AR devices increases risk of eye strain. As such, the literature review will focus on these two aspects.

Neck and Shoulder Muscle Activity

Many studies have shown that patients who experience neck pain have a higher activation of neck flexor and extensor muscles (Johnston et al., 2008; Lindstrøm et al., 2011). In a study conducted on female office workers with and without neck pain, Johnston et al. (2008) found that workers with neck pain relaxed the upper trapezius and cervical extensor muscles at a slower rate than participants without neck pain. In general, workers with mild and moderate neck pain experienced more electromyographic (EMG) activation in the superficial neck flexors than subjects without pain. Therefore, in muscles such as the sternocleidomastoid (SCM) and upper trapezius, a higher level of muscle activation may indicate higher risk of neck pain.

Lindstrøm et al. (2011) characterized the relationship between neck muscle coactivation, neck muscle strength and neck pain. In general, the study found that the subjects with neck pain experienced a higher level of coactivation of the SCM and splenius (SPL) muscles compared to that of the control group. The study also found that a higher level of coactivation of the SPL was associated with lower neck strength and higher levels of neck pain.

Ergonomics studies have tried to decrease the muscle activation of neck flexor and extensor muscles in order to decrease neck pain. Forde et al. (2011) placed a weight on the posterior side of the helmet of a helicopter pilot to counteract the extra weight of night vision goggles. Counterbalanced weights were also used on office workers with the Neck Balance System (NBS; Quinzi et al., 2019), which was a hat with a weight applied over the posterior region of the brain. The NBS reduced the exertion of the semispinalis capitis (SMP) and had no effect on neck flexor muscles. This concept is relevant because a prolonged forward head posture can cause neck pain, stemming from neck muscle activation and spinal compression (Schüldt et al., 1986).

Eye Strain

A major physiological indicator of eye strain is decreased blink rate. Yan et al. (2008) found that eye strain can be measured quantitatively by blink rate. Specifically, if blink rate substantially decreases while using a computer, it may be indicative of eye strain. Other studies have also confirmed this relationship (Abelson & Ousler, 1999; Blehm et al., 2005). In a study of 104 office workers, Tsubota and Nakamori (1993) found that the mean blink rate was 22 blinks/min while relaxed, ten blinks/min while viewing a book, and seven blinks/min while viewing a digital screen.

Objective

The objective was to test the effect of two common AR systems, the HoloLens (first generation) and HMT-1, on muscle activity and eye strain of two electric utility worker groups who perform procedural field work.

Experimental Method

AR Systems

The Microsoft HoloLens (first generation) and RealWear HMT-1 were selected for testing because they are likely to be among the first devices for testing with industrial and utility workers. Both AR devices can be attached to a construction hard hat with plastic hardware.

The HoloLens has a visor with tinted glass through which the user sees the real world, as shown in Figure 1 (left) and Figure 2. The HoloLens, which accommodates users who wear glasses, has digital displays of data and information that are super-imposed on the interior of the visor. The focal distance of the digital displays, which can be static (PowerPoint; PPT) or video, is more than 2 m in front of the eyes (Condino et al., 2020). The weight of the HoloLens was 0.58 kg, and the total weight of the HoloLens with the MSA V-Gard hard hat and attachment hardware (Meemim vGIS, Toronto, Canada, for power plant workers; Trimble, Sunnyvale, CA, for manhole workers) was 1.0 kg.

Figure 1

Left: HoloLens worn by utility worker. Right: RealWear HMT-1 worn by worker. The small display is mounted to the side of the right eye. The worker wore safety glasses, and an eye camera was located under the right pane of the safety glasses.

Figure 2

Plant operator gesturing with “air taps” to communicate with the HoloLens before inspecting equipment.

The HoloLens has the capability for a worker to communicate with a hand gesture (air tap) or speech recognition via voice input. The air tap method, which consists of placing the index finger in front of the HoloLens at arm length and then moving the index finger to the thumb to tap, was used in the present study for both groups of workers (Figure 2). Speech recognition for the first-generation HoloLens was reported to be problematic (Strange, March 12, 2019) and was corroborated with testing in the power plant. For two workers (male and female), 90% of verbal commands to the HoloLens were not transmitted successfully in the power plant, which had a noise level ranging from 81 to 84 dBA.

The RealWear HMT-1 is an AR device that has a small digital screen (0.9 × 1.5 cm) located to the side of one of the user’s eyes and is attached to a hard hat as shown in Figure 1 (right). The distance from the eye to the screen was approximately 6.5 cm, and the angle of the boom arm to the eye was adjusted so the user saw the image on the display in focus. Static slides or videos can be displayed on the small screen. The HMT-1 weighed 0.37 kg, and the total weight of the HMT-1 with the hard hat and attachment hardware was 0.76 kg. The attachment hardware consisted of accessory clips provided by RealWear.

Two utility worker groups were selected for testing: operators who inspect equipment in power plants and manhole workers who splice cable. These workers spend most of their shift doing procedural work and are likely to be initial users of AR in the utility sector.

Power Plant Equipment Inspection Task

The first phase was conducted in a large two-unit coal-fired electric utility power plant located in the Midwestern United States. A major part of a power plant operator’s job is to check the status of over 200 pieces of equipment in a 10 h shift. The information that operators use are the exact equipment and their location and the procedures to follow if a piece of equipment is not functioning. Operators either memorize this information or carry paper instruction sheets. Five inspection tasks were conducted on the following equipment at the coal feeder station in the power plant:

Right and left sight glass to check for coal flow

Calcium bromide piping system

Drive motor

Clean out conveyer motor

Coal feeder side doors

Manhole Cable Splicing Task

An electric utility manhole is an underground confined space that provides access for making connections to and maintaining a network of underground electrical cables. A typical size of manhole conductor is 750 kcmil, which has a metal conductor diameter of 2.2 cm and a total diameter with insulation and taping of 7.6 cm or larger. All electrical splices in a manhole must be waterproof, and a common method to ensure waterproofing is with hand taping (Figure 3). A worker applies many rolls of tape to make the splice waterproof. The entire splicing process is a long, manually intensive operation with multiple steps that may last a total of 45 min or more for one large conductor joint. The taping process requires compliance with precise standards in order to ensure safety to the worker and public and for system reliability. Workers often carry paper sheets with them to the manhole to make sure that their method is in compliance.

Figure 3

Manhole worker hand taping a splice while wearing the HMT-1. A cell phone attached to his right arm recorded data from the eye camera while the worker did the task.

There are three main tasks of splicing: manhole check, splice preparation, and hand taping.

Manhole Inspection. A worker uses a paper layout card for that specific manhole and checks that all the cables are marked correctly. This task is relatively quick, approximately 1 min.

Splice Preparation. The worker strips the insulation off the ends of the cable according to specifications. The duration of this task can be long (>30 min) so up to 10 min of data were collected.

Splice Taping. The worker crimps the two conductors and then hand tapes the entire splice (Figure 3). This task also tends to have a long duration (>30 min) so 5 min of data were collected.

Subjects

All subjects were free of musculoskeletal injury or pain and exhibited full mobility of the neck and upper extremities. A priori analysis of number of subjects to determine statistical power was not conducted because there were no previous studies that reported standard deviations of the dependent variables (EMG and blink rate) for utility or industrial workers who used AR devices. As such, convenience samples of 12 and 13 subjects were tested in both phases, respectively, to establish baseline data. In addition, these samples were the maximum that the utility allowed for testing on-site. All subjects signed a Marquette University-approved consent form before the study commenced.

Power plant operators

Twelve (11 male and one female) power plant operators participated, and their height and weight were 180.8 (±6.8) cm and 105.6 (±20.6) kg, respectively. The mean age was 42 with a range from 27 to 56 years, and the years of experience as an operator averaged 8.2 years with a range from 4 to 18 years. Eleven subjects were right-eye dominant based on the Washburn et al. (1934) method.

Manhole workers

Thirteen male electric utility manhole workers participated, and their mean height and weight were 180.5 (±6.6) cm and 104.2 kg (±15.8), respectively. Mean age was 38 with a range from 25 to 51 years, and their experience as a manhole worker averaged 4.7 years with a range from 3 months to 20 years. Ten subjects were right-eye dominant.

Experimental Design

A repeated measures design was employed to test one independent variable, type of AR device. There were three experimental conditions: HoloLens attached to hard hat, HMT-1 attached to hard hat, and “No AR” (hard hat only: regular method). The dependent variables were the following:

Time (s) to complete each task

Fiftieth and 90th percentage of maximum voluntary contraction (%MVC_EMG) amplitude probability distribution function (APDF) of surface electromyography (sEMG) signals. Fiftieth and 90th APDF represent average and peak values of sEMG activity.

Blink rate of right eye (blinks/s)

Subjective assessment of perceived safety, like/dislike, and comfort of AR device with a Likert scale (1 to 7)

sEMG equipment

Four pairs of muscles were monitored with Biometrics Ltd. (Gwent, UK) integral differential surface EMG sensors (model SX230): sternocleidomastoid (SCM), splenius (SPL), semispinalis capitis (SMP), and upper trapezius (TRAP; Figure 4). Voltage recorded from the bipolar sEMG sensors were communicated to a Biometrics Ltd. Data Logger, which was strapped to the subject’s belt, and these data were transmitted wirelessly to a laptop computer. Biometrics Ltd. data management software recorded and processed the signals and stored the data for subsequent analysis. Specifications of the sEMG sensors were: inter-electrode distance of 20 mm, gain of 1000 with a bandwidth ranging from 20 to 450 Hz, and sampling rate of 1000 Hz.

Figure 4

Left: Locations of sEMG electrodes on the right and left sternocleidomastoid (SCM). Right: Locations of sEMG electrodes on the right and left trapezius (TRAP), splenius (SPL), and semispinalis capitis (SMP; from lateral to medial). The center electrode served as the reference ground (GND), which was located on the first thoracic transverse process.

Eye camera

Blink rate was recorded with the Pupil Labs (Berlin, Germany) 200 Hz Eye Camera. The camera was mounted to the HoloLens with the binocular add-on made by Pupil Labs, and for the HMT-1 and “No AR” conditions, the camera was mounted to a 3D-printed frame made by Pupil Labs. The camera recorded blink rate of the right eye of all subjects, regardless of eye dominance. Blink rate data were stored on a cell phone that was strapped to the subject’s right arm (Figure 3), and the data were subsequently transferred to a PC.

AR Training

Training to use the two AR devices was the same for both the power plant operators and manhole workers. Only the AR training for the plant operators is described in the manuscript to minimize manuscript length. Practice session for each AR device lasted approximately 15 min, which was sufficient to train workers for each AR device.

HoloLens (hand gesture “air tap” communication)

The subject was trained on how to use the HoloLens using the air tap command (Figure 2). This tapping motion represents a command that acts like a mouse click, and the subject practiced air tapping with his right hand without wearing the HoloLens. Then the subject looked at a PPT slide on the investigator’s laptop. This slide showed the flow of content of the subsequent PPT slides that told the worker the process for inspecting equipment or splicing a cable.

The subject then donned the HoloLens with the hard hat and adjusted the HoloLens in order to fix the PPT display in a fixed virtual location near the equipment to be inspected. This process was done with spatial mapping of the physical environment around the equipment using geometric mesh software. When the operator looked directly at the fixed virtual site in the real physical environment, the PPT filled the field of vision of the HoloLens. When they looked away from the fixed site, the PPT slide image was either in the periphery or not in the field of vision. Figure 5 shows a sketch of what an operator would likely see through the HoloLens while inspecting the sight glasses equipment. Sketches are shown because it was not possible to capture a digital image of the view through the HoloLens.

Figure 5

Example of spatial mapping of the PPT slide for Sight Glass Checks inspection task. Top: Plan view of fixed location of sight glass checks PPT slide in the physical environment of sight glass equipment. The operator selected this location for the PPT slide, using air taps to communicate with the HoloLens, so they could see the PPT information in the periphery of the field of vision or with minimal head movement (if the PPT slide was outside field of vision). Bottom: Approximate view that the operator sees through the HoloLens visor with PPT slide in the location selected in the plan view (top).

The subject practiced navigating the PPT slides with the HoloLens until they were comfortable using air taps to click through the slides using the hyperlinks that describe the respective work processes.

HMT-1 (speech communication)

The HMT-1 was attached to a hard hat and adjusted to subject’s head. The boom arm was adjusted so the display did not obstruct the line of sight of the subject’s visual field. The mean declination angle of the boom arm was 30° to the ear-to-eye line. Then the subject confirmed if he was right or left eye dominant by the Mile’s test (Washburn et al., 1934). The HMT-1 was attached to the hard hat on the subject’s dominant eye side. The subject practiced navigating the HMT-1 PPT slides, which had the same content as the HoloLens PPT slides, with the following speech commands:

Next Page: moves to the next page

Previous Page: moves to the previous page

Select Zoom Level 1–5: sets zoom level on document

Freeze and Unfreeze Document

Experimental Protocol

The experimental protocol for both groups of workers was the same, unless otherwise noted. Anthropometric measurements of height and weight were recorded along with a survey of occupational background information. The sEMG system was then set up, and sEMG electrodes were placed on the following eight muscles. Subject exerted 100% MVC with the following postures (Basmajian, 1998):

Sternocleidomastoid (SCM)

Left: Place right palm over right temple and rotate head to right

Right: Place left palm over left temple and rotate head to left

Semispinalis capitis (SMP)

Place both palms on back of head with fingers locked and resist neck extension

Splenius (SPL)

Left: Left palm on left rear of head and resist head rotation to the left

Right: Right palm on right rear of head and resist head rotation to the right

Upper trapezius (TRAP)

Place palms on back of head with fingers locked and shrug (lift) shoulders

One of the three experimental conditions (HoloLens, HMT-1, or “No AR”) was selected from a counter-balanced presentation order to avoid carryover and order effects. The subject donned the corresponding AR device or no AR, and the investigator attached the eye camera to the right side of the head. Then, the tasks commenced in a fixed order for each group of workers.

The operators inspected the five pieces of equipment in a standing posture in the order previously described. The distance between adjacent pieces of equipment was no more than 3 m. The subject walked from one piece of equipment to the next with no rest. Neck and arm posture did not vary between the three methods to do the equipment inspections. Neck flexion posture ranged from 0° to 30° and range of arm flexion was 0° (side of trunk) to 45°. After the five pieces of equipment were inspected, the subject sat down and filled out a subjective assessment survey. The total rest time, which included filling out the survey, was approximately 10 min before the next experimental condition commenced. The protocol for the manhole workers was the same as for operators except there were three tasks (manhole inspection, splice preparation, and taping), which were conducted in that order. Neck and arm posture did not vary between the three methods to do the manhole splicing; neck flexion ranged from 0° to 45° and the arms were flexed and abducted up to 45°. Total test time for each subject was approximately 3 to 4 hr.

Subjective Assessment Questions

After each test session(i.e., five inspection tasks with an AR device, three manhole tasks with an AR device), the subject answered questions on a Likert scale (1 to 7) to assess various attributes of the respective device for conducting those tasks.

Data Conditioning and Analysis

The dependent variable data for the operators and manhole workers were conditioned and analyzed in the same manner, unless otherwise noted.

For both groups of workers, the dependent variable data (time duration, sEMG, and blink rate) during the time that the subject was inspecting the equipment or doing manhole work were analyzed to make a fair comparison of the differences between the two AR devices and “No AR.” For the plant operators and manhole workers, analysis did not include the time for a subject to conduct air taps to communicate with the HoloLens because the time required to make air taps was often longer than the inspection duration. The HoloLens air tap gesture was a discrete task from the equipment inspection and manhole tasks, and thus the air tap time was not included in analysis. In contrast, workers were able to make speech commands to the HMT-1 while they were inspecting the equipment or splicing cable, and it was not feasible to separate speech time from the actual work time.

The number of blinks was counted by Pupil Labs software and later normalized to blinks per minute. Blink rate data from 10 subjects for each worker group were analyzed because the eye camera did not work properly for the remaining subjects. Blink rate data did not include the time to make air tap commands with the HoloLens.

Two trials per task per subject were conducted. Time to complete task, blink rate, and 50th and 90th APDF %MVC_EMG were averaged across both trials to produce mean values per subject for the corresponding experimental condition. Then these subject means were analyzed with ANOVA using Minitab software (State College, PA). The nonparametric Friedman’s test was used to analyze the ordinal Likert subjective data because Likert data are discrete and may not follow parametric assumptions.

Results

Power Plant: Environmental Conditions

Ambient temperature averaged 25.3°C (±2.7) with a range of humidity from 13% to 59%, and noise averaged 82.5 dBA (±1.0). Illumination ranged from 12 to 186 lux across the inspection tasks.

Power Plant: Task Duration

The inspection tasks ranged from an average of 10 to 28 s (Table 1). For task 2, the “No AR” condition required about 2.5 s less time than the two AR devices. For task 5, the “No AR” condition required 2.7 s less time than the HoloLens.

Table 1

Average Time (SD) in Seconds to Complete Each Inspection Task With the Two AR Devices and “No AR” (N = 12)

	HoloLens Time (in s)	HMT-1 Time (in s)	“No AR” Time (in s)
Coal Feeder 1	23.7 (13.3)	27.4 (18.0)	23.5 (14.6)
Coal Feeder 2	13.0^a (2.3)	13.5^a (3.7)	10.7^b (3.0)
Coal Feeder 3	23.3 (6.3)	22.7 (5.4)	22.5 (5.8)
Coal Feeder 4	15.3 (7.8)	14.4 (6.6)	14.1 (7.9)
Coal Feeder 5	15.6^c (6.9)	14.2^c,d (6.7)	12.9^d (7.3)

Note. ^a,bMeans that do not share a letter are significantly different (p < 0.05). ^c,dMeans that do not share a letter are significantly different (p < 0.05).

Power Plant: 50th and 90th Percentile sEMG

Except for one case (90th percentile left SMP for coal feeder task 3), there were no significant differences for the left- and right-side 50th and 90th percentile %MVC_EMG between the three experimental conditions. Only the right-side muscles will be described because of the similarity in sEMG results between the two sides. The 50th percentile MVC_EMG of the right muscles ranged from approximately 5% to 20%. The right SCM generally had the lowest 50th percentile muscle activity (up to 10% MVC_EMG), while the right SMP had the greatest activity (up to 20% MVC_EMG). The mean 90th percentile EMG activity of the right SCM ranged from ranged between 10 and 20% MVC_EMG while the right SMP was within a range of 13% to 43% MVC_EMG.

Power Plant: Blink Rate

The blink rate was computed for three of the five inspection tasks because these tasks had sufficient time duration (at least 11 s) to assess the number of blinks/min reliably (Table 2). The average blink rate for the HMT-1 and “No AR” conditions ranged from 15 to 18 blinks/min, and the mean HoloLens blink rate was 10 to 13 blinks/min. The differences in means were not statistically significant (p = 0.06 to 0.17).

Table 2

Average Blink Rate (SD) for Each Combination of Experimental Condition and Inspection Task (N = 10)

	Average (SD) Blink Rate (Blinks/Min)	HoloLens Blink Rate Decrease(Blinks/Min & %^a)	p Value^b
Coal Feeder 1 Inspection Task
HoloLens	12.5 (6.2)	3.8 (23.3%)	0.173
HMT-1	18.0 (8.7)
No AR	16.3 (7.3)
Coal Feeder 3 Inspection Task
HoloLens	10.9 (5.8)	4.4 (28.8%)	0.170
HMT-1	15.3 (6.4)
No AR	15.3 (6.5)
Coal Feeder 4 Inspection Task
HoloLens	10.7 (4.3)	5.1 (32.2%)	0.062
HMT-1	15.0 (6.5)
No AR	15.8 (7.9)

Note. ^aPercentage decrease relative to the “No AR” condition. ^bp value of main effect from ANOVA.

Power Plant: Subjective Assessment

The HMT-1 was rated easier to use than the HoloLens, with medians of 6.5 and 5, respectively (p = .02), on a scale of 1 (very difficult) to 7 (very easy; Question 1, Table 3). The HMT-1 was also rated more comfortable than the HoloLens, with medians of 6.5 and 5, respectively (p = .01; Question 2, 1: very uncomfortable; 7: very comfortable). The ratings of dislike/like of each AR device (Question 7) were widespread and not significantly different, with the HoloLens ratings ranging from 1 to 6 and the HMT-1 ranging from1 to 7 (1: dislike very much; 7: like very much). Mean ratings of situational awareness of objects around the subject (Question 6) did not vary significantly (p = .15; 1: very difficult to see things around me; 7: very easy to see things around me).

Table 3

Summary Statistics of Plant Operators’ Subjective Assessment Responses (N = 12)

	Question	HoloLens	HMT-1
Coal Feeder Equipment Inspection Tasks	Q1: Ease of use
	Average	4.6^a	6.4^a
	Median	5	6.5
	SD	1.3	0.7
	Range	3–6	5–7
	Q2: Comfort/discomfort
	Average	3.7^b	5.5^b
	Median	3.0	6.0
	SD	1.6	1.5
	Range	1–6	2–7
	Q3: Effect on eyes
	Average	6.2	6.3
	Median	6.5	7.0
	SD	1.0	1.0
	Range	4–7	4–7
	Q4: Effect on neck and shoulder muscles
	Average	6.3	6.8
	Median	6.5	7.0
	SD	1.0	0.4
	Range	4–7	6–7
	Q5: Effect on safe equipment inspections
	Average	5.9	6.4
	Median	6.0	7.0
	SD	1.2	1.2
	Range	3–7	3–7
	Q6: Effect on your awareness of things and objects around you
	Average	4.7	5.4
	Median	5.0	6.0
	SD	1.9	1.5
	Range	2–7	3–7
	Q7: Like/dislike of the device for equipment inspections
	Average	3.9	4.8
	Median	4.0	6.0
	SD	2.1	1.8
	Range	1–6	1–7

Note. Each question had Likert responses ranging from 1 to 7, which corresponded from least favorable to most favorable attributes, respectively. Differences between means for the HoloLens and HMT-1 were significant for ease of use (Q1, p = .021^a) and comfort (Q2, p = .009^b).

Manhole: Environmental Conditions

Ambient temperature averaged 13.8°C (±7.3) with a range of relative humidity from 13% to 40%, and mean noise level was approximately 70 dBA. The lighting levels varied inside the manhole due to portable lighting fixtures that were installed to illuminate the cable joint. High illumination from the lights was focused on the splicing areas (1293 to 1477 lux) while the darker, unlit areas in the manhole ranged from 304 to 331 lux.

Manhole: Task Duration

The average times to complete the three tasks ranged from less than 1 min (manhole inspection) to almost 10 min (splice preparation), and there were no significant differences to complete the tasks between the AR and “No AR” conditions (Table 4). The splice preparation and hand taping test session times were stopped early by the experimenter (10- and 5-min marks, respectively) to make sure the entire study time was feasible for workers and the utility. More time was given to the preparation task because this task had more steps.

Table 4

Summary Statistics of Time (S) to Complete Manhole Tasks (N = 13)

	HoloLens	HMT-1	“No AR”
Manhole Inspection
Mean	51.5 s	50.2 s	45.8 s
SD	30.8	25.4	23.8
Splice Preparation
Mean	547.2 s	522.0 s	554.9 s
SD	137.4	193.2	169.4
Splice Taping
Mean	306.7 s	292.3 s	281.7 s
SD	87.8	112.4	72.4

Note. There were no significant differences between means.

Manhole: 50th and 90th Percentile EMG

Similar to the power plant workers, there were generally no significant differences for the 50th and 90th percentile %MVC_EMG of the left and right muscles between the three experimental conditions within each task (Figure 6).

Figure 6

Means + SD (N = 11 to 12) of the 50th percentile %MVC_EMG for the left and right SCM, SMP, SPL, and TRAP muscles for the three manhole tasks. There were no significant differences between AR and “No AR” conditions except for the left SCM in the inspection task (“No AR” > HMT-1), the right SMP in the inspection task (HoloLens > HMT-1), and left TRAP in the preparation task (“No AR” > HoloLens).

Manhole: Blink Rate

The average blink rate for the HMT-1 and “No AR” conditions ranged from 15 to 22 blinks/min in the inspection and taping tasks while the HoloLens blink rate ranged from 7 to 14 blinks/min (Table 5). There were no differences between the HMT-1 and “No AR” conditions for all three tasks, but the HoloLens blink rate was significantly lower than the HMT-1 conditions by 7.8 and 10.9 blinks/min in two tasks, inspection (p = .027) and taping (p = .035). The HoloLens blink rate was also 6.5 blinks/min lower in the inspection task compared to “No AR” (p = .007). There were no differences in blink rate between the experimental conditions for the preparation task, with the mean blink rate ranging from 10.8 to 14.7.

Table 5

Mean Blink Rate (SD) for Each Combination of Experimental Condition and Manhole Task

	Mean (SD) Blink Rate (Blinks/Min)	HoloLens Blink Rate Decrease to HMT-1(Blinks/Min & Percentage)	HoloLens Blink Rate Decrease to “No AR”(Blinks/Min & Percentage)	p Value FromANOVA^a
Inspection Task (N = 10)
HoloLens	14.1 (5.6)	7.78 (35.6%) p = .027^b	6.54 (31.6%)p = .007^b	.015
HMT-1	21.9 (11.2)
“No AR”	20.6 (6.4)
Preparation Task (N = 9)
HoloLens	10.8 (4.4)	2.76 (20.6%) p = .144	3.86 (26.5%)p = .110	.102
HMT-1	13.6 (4.9)
“No AR”	14.7 (5.6)
Taping Task (N = 10)
HoloLens	6.9 (5.2)	10.91 (61.2%) p = .035^b	7.89 (53.1%)p = .054^b	.018
HMT-1	17.8 (12.3)
“No AR”	14.7 (9.2)

Note. Sample size was fewer than N = 13 because the eye camera did not work for some subjects because of obstructions. ^aRepeated measures ANOVA. ^bp value from post-hoc paired t test.

Manhole: Subjective Assessment

As revealed in Table 6, mean ratings of the HoloLens and HMT-1 according to various attributes measured with a Likert scale of 1 to 7 were close and not significantly different except for comfort (the HMT-1 was rated more comfortable (4.5 vs. 3.6, p = .027). Of note are the average ratings of dislike/like of AR devices; the median ratings of the HoloLens and HMT-1 were identical (5) with the same ranges (2 to 7).

Table 6

Summary Statistics of Subjective Assessments of the AR Devices for Conducting Manhole Tasks (N = 13)

Question	Rating
Question	HoloLens	HMT-1
Q1: Ease of use
Mean	4.5	5.1
Median	4.0	5.5
SD	1.3	1.2
Range	3–7	3–6
Q2: Comfort/discomfort
Mean	3.6^a	4.5^a
Median	3.0	4.5
SD	1.3	1.5
Range	2–6	3–7
Q3: Effect on eyes
Mean	5.5	5.9
Median	6.0	6.0
SD	1.3	0.9
Range	2–7	5–7
Q4: Effect on neck and shoulder muscles
Mean	5.7	6.3
Median	6.0	7.0
SD	1.3	1.0
Range	2–7	4–7
Q5: Effect on safe equipment inspections
Mean	6.2	6.5
Median	7.0	7.0
SD	1.3	0.9
Range	3–7	4–7
Q6: Effect on your awareness of things and objects around you
Mean	5.0	5.0
Median	6.0	5.5
SD	1.5	1.8
Range	2–7	3–7
Q7: Dislike/like of the device for manhole work
Mean	4.9	4.5
Median	5.0	5.0
SD	1.3	1.4
Range	2–7	2–7

Note. Questions had a Likert scale from 1 to 7 (least favorable to most favorable, respectively). ^aMeans significantly different (p = .027).

When asked about their views of the AR devices for manhole work (Table 7), none of the 13 subjects wanted to use the HoloLens for more than 4 h per day but five said they would use it for less than 4 h per shift. The ratings for the HMT-1 were higher, with two subjects responding “yes” for more than 4 h and eight for up to 4 h. The workers thought the AR devices had potential for training, with 11 votes for “yes” or “maybe” for the HoloLens and 13 for the HMT-1.

Table 7

Summary Statistics of Subjective Ratings of the AR Devices for Total Job as Manhole Worker (N = 13)

	HoloLens		HMT-1
	Number	Percentage	Number	Percentage
Q11: Would you use the AR device on the job MOST of the TIME (4 h or more)?
Yes	0	0.0%	2	15.4%
No	8	61.5%	4	30.8%
Maybe	5	38.5%	7	53.9%
Q12: Would you use the AR device on the job SOMETIMES (1 to 4 h)?
Yes	5	38.5%	8	61.5%
No	1	7.7%	0	0.0%
Maybe	7	53.9%	5	38.5%
Q14: Do you think the AR device may help underground workers in their training?
Yes	5	38.5%	5	38.5%
No	2	15.4%	0	0.0%
Maybe	6	46.2%	8	61.5%

Workers did not rate situational awareness differently between the HMT-1 and HoloLens (Q5 and Q6 in Table 6). Subjects reported that they were able to operate equipment with both AR devices as safely as under normal conditions (median scores of 7 for Q5). Median ratings of being aware of things in the environment were “easy to see things” (median of 6) for the HoloLens and between “somewhat easy” and “easy” for the HMT-1 (median of 5.5).

Written comments from workers revealed that the HoloLens and HMT-1 would be useful for tasks that involved charting, specifications for splices, and real-time updating of specifications and other information. Some subjects reported the HoloLens as too heavy, cumbersome, and hard to read if lighting is not right. There were fewer negative comments about the HMT-1, and some of the positive comments were that it was lightweight, easy to use, and less eye fatiguing to focus on details in the environment than the HoloLens.

Discussion

Task Duration of Utility Jobs

This study analyzed two utility jobs that had a wide range of task duration. Task time for plant operators performing equipment inspections was short (<30 s), while time to perform complex tasks of preparing and taping splices in a manhole can range up to 45 min each. This large range of time duration encompasses most electric utility field work and thus provides insight about the effect of AR devices on utility field work.

Communication Methods With AR Devices

Subjects used speech to communicate with the HMT-1 but employed the air tap hand gesture to communicate with the HoloLens. The first-generation HoloLens was not able to recognize reliably workers’ speech in the noisy power plant (>81 dBA). Microsoft acknowledged the speech recognition deficiency for the HoloLens and reported that it will likely be resolved with future versions (Strange, March 12, 2019). To ensure consistent comparison with power plant data, subjects used the air tap method with the HoloLens in the manhole, which had a less noisy environment.

The time to conduct air taps with the HoloLens was not included in collection and analysis of time duration, sEMG, and blink rate data because it is expected that speech recognition will be feasible in noisy environments with later generations of the HoloLens, and thus, data from future studies of the HoloLens can be compared to the present study. We believe that the HoloLens data from the present study (without the air tap gesture) will serve as a good, but not perfect, baseline to inform future studies of the HoloLens that use speech recognition.

Unlike intervals of air tap commands, speech commands generally are concurrent with physical work and are more integrated into a task than discrete air tap hand gestures. For this reason, it was not possible to separate time intervals between speech commands and physical work when workers used the HMT-1.

Task Duration Time

There was no difference in time to perform plant inspections between the HoloLens and HMT-1, but the normal method (“No AR”) was about 2.5 s less than both AR devices in two of the five inspections. For the manhole operations, it was not possible to assess accurately differences in task time between AR and “No AR” conditions because test time was stopped early at the 10- and 5 min marks for the preparation and hand taping tasks, respectively, to limit the total study time to less than 4 h per subject.

sEMG Muscle Activity

In general, sEMG activity of the neck and upper shoulder muscles did not vary for both equipment inspections and manhole work, despite the fact that the HoloLens and HMT-1 were heavier than the “No AR” condition (0.58 and 0.37 kg, respectively; Figure 6). Fiftieth percentile EMG activity of the three major neck muscles (SCM, SMP, and SPL) ranged from 5% to 15% MVC, which appears reasonable because the workers were moving their head up and down to inspect equipment and prepare and splice cable. The average %MVC of neck muscles of office workers with minimal head movement was about 4.5% MVC (Quinzi et al., 2019).

Due to the small sample size and short testing times, the study was not able to confirm whether the extra weight of the HoloLens and HMT-1 increased muscle activity compared to the absence of AR. Future studies should increase the sample size to test for small differences in sEMG. Industrial and field workers are likely to wear AR devices for long durations during the day, probably at least 2 to 3 h, and thus, there may be an increase in muscle activity in the neck and upper shoulder muscles during long-duration work.

Blink Rate

The HoloLens blink rate for the manhole inspection and taping tasks was 8 to 11 blinks/min fewer than the HMT-1 (Table 5). Based on the literature, this magnitude of reduction may present risk of eye strain to manhole workers who use an OST AR device for long-duration procedural tasks. In a study of people reading a hard copy book and an electronic tablet, the average blink rate decreased 8 and 5 blinks/min, respectively, compared to a relaxed state (Abusharha, 2017). Subjects reported a higher level of ocular discomfort while reading than in the relaxed state. An OST AR device, such as the HoloLens, presents information in digital displays in the worker’s field of view, and thus it may require a level of visual focus that is similar to reading hard copy or viewing a digital device.

A lower blink rate can result in eye fatigue and dry-eye symptoms, which commonly affect people with computer vision syndrome (Rosenfeld, 2016; Yan et al., 2008). Cardona et al. (2011) found that a lower blink rate can result in excessive tear film evaporation from the ocular surface and can lead to visual fatigue and ocular discomfort. Furthermore, an environment with varying brightness levels can cause visual discomfort due to transient adaption from fixating back and forth between two illumination levels (Yan et al., 2008). The lighting levels in the power plant varied from 12 to 186 lux for equipment inspections and from 300 to 1300 lux in the manhole. Visual discomfort, and possibly dry-eye symptoms, may result if utility workers were to use AR OST devices in environments of varying illumination for long-duration procedural tasks that require sustained visual attention.

Another issue that can affect eye strain of workers using AR OST devices is the vergence-accommodation conflict and focal rivalry (Condino et al., 2020). If a virtual image overlays a real object in the field of view, then focal rivalry can result due to the difference in focal lengths between the two overlapping images. The focal length of the HoloLens was at least 2 m (Condino et al., 2020), and if tasks were to require overlay of virtual and real objects of interest, then focal rivalry may result and cause eye strain. When the HoloLens was used in the power plant and manhole studies, the virtual PPT images did not overlay directly the real objects pertinent to the task (inspected equipment and cables for splicing) but were presented on the periphery of the intended visual target (Figure 5). Likewise, the PPT slides for the HMT-1 were located to the side of the face and in the periphery of the field of vision. Whether the spatial relationship between the work objects and PPT slides seen through the HoloLens or with the HMT-1 would create a vergence-accommodation conflict or focal rivalry is not known at this time.

Subjective Assessment and Situational Awareness

Subjective assessments of the AR devices were consistent for both plant operators and manhole workers. Both groups rated the HMT-1 more comfortable than the HoloLens, and the plant operators rated the HMT-1 easier to use (Tables 3 and 6). The increase in comfort of the HMT-1 over the HoloLens was probably due to the lower weight and lower impact on the worker’s field of view of the HMT-1’s smaller display. The HMT-1 has a small display to the side of the face, in contrast to the visor of the HoloLens that covers the entire field of vision. Both groups of utility workers did not rate situational awareness differently between the HMT-1 and HoloLens (Q5 and Q6 in Tables 3 and 6). Whether this lack of a difference is generalizable to utility workers wearing HMD AR devices for long periods is not known.

AR Potential for Training

Written comments from plant operators and manhole workers indicated the potential of both AR devices for training new workers. Apprentices in both worker groups need to develop an extensive knowledge base of work practices, and they often refer to paper instruction sheets. AR devices can deliver timely and relevant information in digital form in the worker’s field of vision, which may reduce training time.

Limitations and Future Work

We stopped the data collection for the manhole preparation and taping tasks at the 10- and 5-min marks, respectively, to limit the total study time to less than 4 h to make testing feasible for workers and the host utility. However, future studies should strive to collect data for longer periods. Longer test durations with a larger sample size would, in theory, enhance the generalizability of the data to the actual tasks, particularly if workers were to wear HMD AR devices for long periods during a shift.

While workers used speech commands to communicate with the HMT-1, workers communicated with the HoloLens via an air tap hand gesture because the high noise level in the power plant interfered with speech commands. It is expected that future versions of the HoloLens will enable speech communication in noisy environments, so future studies should test the HoloLens with speech communication.

Compared to the HMT-1, the HoloLens decreased blink rate 8 to 11 blinks/min in two of the three manhole tasks. A reduction of these magnitudes may be large enough to indicate the risk of eye strain. Empirical results from the manhole workers along with findings from the literature suggest that, at minimum, future studies should focus on the potential for eye strain from OST AR devices intended for workers who will use them for long-duration procedural work.

The possible effects of OST AR on visual strain of manhole workers may not be generalizable to other types of industrial or field workers. Thus, future studies should test specific work groups and their tasks before OST AR deployment. The placement of the mobile phone on the right arm may have affected the sEMG of the upper trapezius muscle, and thus future studies should investigate other recording technology that does not add weight to the arms.

The cost of HMD AR devices is expensive (at least $1000 for the HMT-1 and more for the HoloLens), and the first round of deployment will likely be targeted to the higher paid workers. Field workers are a prime work sector for HMD AR deployment because these workers tend to be higher paid and the high cost of HMD AR technology may have a short payback period, assuming an increase in productivity with AR. The effect of HMD AR devices on long-term occupational health of workers, of which muscle fatigue and eye strain are two principal issues, is not known. A deep understanding of the long-term occupational health effects of HMD AR technology is needed before deployment to workers.

Conclusions

The MS HoloLens may expose utility manhole workers to risk of eye strain. Compared to the HMT-1, the HoloLens decreased blink rate by 8 to 11 blinks/min in two of the three tasks. A decrease of this magnitude may present risk of eye strain if workers were to use OST AR devices for long durations without appropriate rest periods. Regardless of the brand or model, all AR devices should be thoroughly tested for the specific work group and tasks before they are deployed to workers. The study was not able to confirm whether the extra weight of the HoloLens and HMT-1 increased sEMG activity of the neck and upper shoulder muscles.

Key Points

Two common head-mounted AR devices, the MS HoloLens and the RealWear HMT-1, were tested with electric utility operators and manhole workers. Workers communicated with the HoloLens via a hand gesture and with the HMT-1 via speech input.

With the small sample and relatively short testing times, the study could not confirm whether extra weight of the weight of the HoloLens and HMT-1 increased sEMG activity of the neck and upper shoulder muscles.

The HoloLens blink rate was 8 to 11 blinks/min lower than the HMT-1 in two manhole worker tasks. This finding may indicate a risk of eye strain to utility manhole workers.

Footnotes

Acknowledgments

EPRI’s (Electric Power Research Institute) Occupational Health and Safety Program and the Program on Technology Innovation funded this project. EPRI’s Information and Technology Communications Program provided technical support (contract #10008884). Witor Eng provided technical assistance. The EPRI Occupational Health and Safety Program and Program on Technical Innovation Program funded the project.

Author Biographies

Richard W. Marklin, Jr., is a professor of mechanical and biomedical engineering at Marquette University, Milwaukee, WI. He specializes in physical ergonomics and design visualization techniques for engineering education.

Ashley M. Toll completed her MS in biomedical engineering at Marquette University in 2019. Her research specialty is physical ergonomics related to industrial and field work. She works as an ergonomist for Milwaukee Tool Co.

Eric H. Bauman is program director of Occupational Health and Safety at Electric Power Research Institute (EPRI). His background is in risk management and communications, environmental health and safety (EH&S) management and policy, and regulatory affairs in the public and private sector.

John J. Simmins is Associate Provost for Research and Economic Development at Alfred University. During the time of this study, he was the program director of Information and Technology Communication at EPRI.

John F. LaDisa, Jr., is an associate professor of biomedical engineering at Marquette University, Milwaukee, WI. He was a postdoctoral scholar at Stanford University for two years after earning his PhD in Biomedical Engineering. He directs MARVL - the MARquette Visualization Laboratory.

Robert Cooper is an assistant professor of biomedical engineering at Marquette University, Milwaukee, WI. His specialty is vision human factors.

References

Abelson

M. B.

Ousler

G. W

. (1999). How to fight computer vision syndrome. Review of ophthalmology, 6, 114–116.

Abusharha

A. A

. (2017). Changes in blink rate and ocular symptoms during different reading tasks. Clinical Optometry, 9, 133–138.doi:10.2147/OPTO.S142718

30214369

Basmajian

J. V

. (1998). Primary anatomy. Stipes Publishing.

Blehm

Vishnu

Khattak

Mitra

Yee

R. W

. (2005). Computer vision syndrome: A review. Survey of Ophthalmology, 50, 253–262.doi:10.1016/j.survophthal.2005.02.008

15850814

Braly

A. M.

Nuernberger

Kim

S. Y

. (2019). Augmented reality improves procedural work on an international space station science instrument. Human Factors, 61, 866–878.

Cardona

García

Serés

Vilaseca

Gispets

. (2011). Blink rate, blink amplitude, and tear film integrity during dynamic visual display terminal tasks. Current Eye Research, 36, 190–197.doi:10.3109/02713683.2010.544442

21275516

Condino

Carbone

Piazza

Ferrari

. (2020). Perceptual limits of optical see-through visors for augmented reality guidance of manual tasks. IEEE Transactions on Biomedical Engineering, 67, 411–419.doi:10.1109/TBME.2019.2914517

31059421

EPRI . (2018). Program on technical innovation–literature review of human factors issues in the electric power industry. EPRI Technical Update 3002012532. E. Bauman and J. Simmins, Program Managers.

Forde

K. A.

Albert

W. J.

Harrison

M. F.

Neary

J. P.

Croll

Callaghan

J. P

. (2011). Neck loads and posture exposure of helicopter pilots during simulated day and night flights. International Journal of Industrial Ergonomics, 41, 128–135.doi:10.1016/j.ergon.2011.01.001

10.

Henderson

S. J.

Feiner

. (2011). Augmented reality in the psychomotor phase of a procedural task [Symposium]. Proceedings of the 10th IEEE International Symposium on Mixed and Augmented Reality, New York, NY, IEEE, 135–144.

11.

Johnston

Jull

Souvlis

Jimmieson

N. L

. (2008). Neck movement and muscle activity characteristics in female office workers with neck pain. Spine, 33, 555–563.doi:10.1097/BRS.0b013e3181657d0d

18317202

12.

Lindstrøm

Schomacher

Farina

Rechter

Falla

. (2011). Association between neck muscle coactivation, pain, and strength in women with neck pain. Manual Therapy, 16, 80–86.doi:10.1016/j.math.2010.07.006

20696610

13.

Porter

M. E.

Heppelmann

J. E

. (2017). A manager’s guide to augmented reality. Harvard Business Review, 95, 45–57.

14.

Quinzi

Scalia

Giombini

Di Cagno

Pigozzi

Casasco

Macaluso

. (2019). The effect of an orthotic device for balancing neck muscles during daily office tasks. Human Factors, 61, 722–735.doi:10.1177/0018720818814957

30608175

15.

Rosenfeld

. (2016). Computer vision syndrome (a.k.a. digital eye strain). Optometry in Practice, 17, 1–10.

16.

Schüldt

Ekholm

Harms-Ringdahl

Németh

Arborelius

U. P

. (1986). Effects of changes in sitting work posture on static neck and shoulder muscle activity. Ergonomics, 29, 1525–1537.doi:10.1080/00140138608967266

3816746

17.

Strange

. (March 12, 2019). Microsoft’s HoloLens 2 Team answers more questions about biometric security, audio, & hand tracking. New Reality. https://hololens.reality.news/news/microsofts-hololens-2-team-answers-more-questions-about-biometric-security-audio-hand-tracking-0194712/

18.

Tang

A. O.

Biocca

Mou

. (2003). Comparative effectiveness of augmented reality in object assembly [Conference session]. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, New York, NY, ACM, 73–80.

19.

Tsubota

Nakamori

. (1993). Dry eyes and video display terminals. New England Journal of Medicine, 328, 584–585.doi:10.1056/NEJM199302253280817

8426634

20.

Washburn

M. F.

Faison

Scott

. (1934). A comparison between the Miles A-B-C method and retinal rivalry as tests of ocular dominance. The American Journal of Psychology, 46, 633–636.doi:10.2307/1415504

21.

Wiedenmaier

Oehme

Schmidt

Luczak

Olaf

Ludger

Holger

Reality

. (2003). Augmented reality (AR) for assembly processes design and experimental evaluation. International Journal of Human-Computer Interaction, 16, 497–514.doi:10.1207/S15327590IJHC1603_7

22.

Yan

Chen

. (2008). Computer vision syndrome: A widely spreading but largely unknown epidemic among computer users. Computers in Human Behavior, 24, 2026–2042.doi:10.1016/j.chb.2007.09.004