Choice of Human–Computer Interaction Mode in Stroke Rehabilitation

Abstract

Background and Objective. Advances in technology are providing new forms of human–computer interaction. The current study examined one form of human–computer interaction, augmented reality (AR), whereby subjects train in the real-world workspace with virtual objects projected by the computer. Motor performances were compared with those obtained while subjects used a traditional human–computer interaction, that is, a personal computer (PC) with a mouse. Methods. Patients used goal-directed arm movements to play AR and PC versions of the Fruit Ninja video game. The 2 versions required the same arm movements to control the game but had different cognitive demands. With AR, the game was projected onto the desktop, where subjects viewed the game plus their arm movements simultaneously, in the same visual coordinate space. In the PC version, subjects used the same arm movements but viewed the game by looking up at a computer monitor. Results. Among 18 patients with chronic hemiparesis after stroke, the AR game was associated with 21% higher game scores (P = .0001), 19% faster reaching times (P = .0001), and 15% less movement variability (P = .0068), as compared to the PC game. Correlations between game score and arm motor status were stronger with the AR version. Conclusions. Motor performances during the AR game were superior to those during the PC game. This result is due in part to the greater cognitive demands imposed by the PC game, a feature problematic for some patients but clinically useful for others. Mode of human–computer interface influences rehabilitation therapy demands and can be individualized for patients.

Introduction

Computer-assisted technology is being increasingly used to enhance physical and occupational therapy after stroke.^1,2 Many current applications rely on a traditional human–computer interaction, in which a person uses a game controller such as a mouse with a desktop personal computer (PC) while looking up at a computer monitor (Figure 1A).^3-5 There are potential limitations with this type of human–computer interaction, for example, a subject must perform a visuospatial transform from the coordinates of arm movement space to the coordinates of computer monitor space, and interaction with the computer’s output is indirect.

Figure 1.

A patient playing the (A) personal computer (PC) and (B) the augmented reality (AR) versions of the Fruit Ninja game. Game features and movement demands were identical across the 2 versions of the game. Both required that the subject move the stroke-affected hand across the tabletop in order to control the movement of a cursor that earned points by slicing fruit targets. Both versions of the game leveraged the same camera; however, in the PC version, game activity was displayed using a computer monitor, while in the AR version, game activity was displayed on the tabletop using a projector. This difference imposed 2 key differences. First, during PC play but not AR play, the subject needed to perform an extra spatial transform in order to convert the coordinates of tabletop arm movements to the coordinates of the cursor movements seen on the monitor. Second, during PC play, eye and hand movements were uncoupled, with subjects receiving proprioceptive feedback but no visual feedback (gaze was directed at the monitor not their hand), while during AR play, visual and proprioceptive feedback were coupled. These differences likely account for the significantly poorer motor performances seen with PC play.

Other human–computer interfaces^6-9 have been studied, such as spatial augmented reality (AR; Figure 1B). ¹⁰ With an AR-based approach, subjects interact in the real-world workspace with virtual objects projected by a computer. ¹¹ No visuospatial transform by the subject is required due to the fact that the subject interacts directly with the computer’s output. The extent to which a subject’s interaction with computer output is direct versus indirect may be important at many levels.^12-15 With an indirect interaction, subjects do not handle the computer’s output medium; with a direct interaction, they do. For instance, using a mouse to play a game is an indirect interaction between the user and the output medium (computer monitor), whereas using a finger on an iPad surface or on a virtual object projected by a computer in order to control a game is a direct interaction between user and computer output.

These differences between a PC approach and an AR approach provide an opportunity to study how choice of human–computer interface affects motor performance, and ultimately stroke rehabilitation. To examine this issue, motor performance of patients with chronic stroke was measured as each played 2 different versions of a popular video game, one PC-based and one AR-based. The 2 versions required the same arm movement kinematics to control the game but differed in level of cognitive demand, which was higher with the PC approach because it uncouples a subject’s eye and hand movements and requires the subject to perform a visuospatial transform.^16,17 Increased cognitive demand is an important consideration when designing computerized therapies to promote stroke rehabilitation.^18-20 The current study hypothesized that the higher level of cognitive demand during the PC-based game would be associated with lower game scores and less efficient movements as compared to the AR-based game.

Methods

Subjects

Patients with chronic stroke were recruited from the community. This study was approved by the institutional review board, and all subjects signed informed consent. Entry criteria were age >18 years, history of stroke with onset >6 months prior, and arm weakness operationally defined as Fugl-Meyer (FM) total arm motor score ²¹ of 15 to 55 (out of 66) plus at least 5° active range of motion in either wrist or index finger metacarpophalangeal joint on the stroke-affected side. Exclusion criteria were severe cognitive, attentional, or language deficits; a nonstroke diagnosis that affected arm or hand function; and coexistent major neurological or psychiatric disease.

After signing consent, medical history was reviewed, followed by a brief exam that included a measure of loss of body/function, the FM arm motor scale, ²¹ as well as a measure of activity limitations, the Box & Blocks Test (BBT).^22,23 Subjects then played the PC-based and the AR-based versions of the game with the paretic arm, as below.

Game Design

Design of the computerized PC and AR games was modeled after the movements required by the BBT,^22,23 which asks subjects to coordinate proximal and distal arm activity during a rapidly repeated task with high movement precision. We applied to this framework a simple version of the popular Fruit Ninja game, which by 2012 had been downloaded to 1 in 3 US iPhones. ²⁴ Thus, in both our PC and AR versions of this game, subjects were seated at a table and asked to perform precise, rapid, repeated goal-directed reaching tasks requiring coordinated hand/shoulder movements (Figure 1). A small plastic cup was held in the stroke-affected hand. This cup served as a color-marker that was tracked by the camera and guided the fruit-slicing cursor during game play; during the PC game this cup guided the on-screen cursor, while during the AR game, this cup was the cursor.

Subjects were seated and introduced to the 2 versions of the game. A fruit target would then appear in 1 of the 4 corners of the tabletop and start moving around the desktop workspace. The subject was to use the stroke-affected hand to move the cursor onto the fruit (which would slice the fruit in half, make a noise, and score 1 point) before the fruit disappeared (which would score 0 points). Once the fruit was sliced, or disappeared unsliced, a new fruit target would appear in 1 of the 4 corners, the sequence of which was pseudorandomized. Subjects were instructed to slice as many fruits as possible within the allotted time. The score was presented throughout each game. One round of each game was 90 seconds long.

Subjects played one practice round each of the PC game and the AR game, which were not recorded or analyzed. Next, study data were acquired as the subject played 3 rounds of either the PC or the AR game, the order of which was randomly assigned across subjects, followed by 3 rounds of the other version of the game. Subjects were provided with a brief break between each round.

PC Versus AR Game Setup

All game features and arm movement demands were identical across the 2 versions of the game and required that the subject move the stroke-affected hand around the tabletop to slice fruit. Subjects sat in the same chair at the same table. Both game versions used the same overhead camera (Microsoft LifeCam VX-2000 Webcam, 30 fps, 720p HD capture) to capture the subject’s tabletop hand movements, which moved the game’s fruit-slicing cursor. These camera images were fed to a computer running a program that tracked the color-marker and that provided real-time audiovisual feedback.

PC Setup (Figure 1A)

The subject was instructed to gaze straight ahead at a 15-inch computer monitor whose position was fixed throughout the experiments. This monitor displayed the fruit targets as well as the cursor, a circle that represented the real-time position of the subject’s hand and moving in a 15-inch tabletop workspace. With the PC setup, the subject’s hand was 45° away from his/her direction of gaze.

AR Setup (Figure 1B)

The subject was instructed to gaze directly at the tabletop, onto which images appeared from a projector (P4-X Pico Projector, AAXA Technologies, Tustin, CA); the monitor was not used and remained blank. The projector displayed (720p HD) fruit targets, and the 15-inch workspace with 4 corners, directly onto the tabletop. With the AR setup, the subject gazed directly at his/her hand as it moved to slice the virtual fruit targets.

Performance Metrics

The primary metric was game score. Four secondary metrics were extracted from game performance to aid interpretation of game scores:

1. Game score: The number of fruit targets successfully sliced during a 90-second round of game play. The 3 PC rounds were averaged to produce a single score, as were the 3 AR rounds.

2. Reaching time: The average amount of time between reaching one fruit target and reaching the next fruit target.

3. Movement time consistency: The coefficient of variation of reaching times.

4. Response latency: The amount of time between display and target reaching, recorded for the first fruit target only in each round of the game.

5. Movement efficiency: The ratio of the ideal travel distance (the distance that would be travelled if the same number of targets were reached using perfect straight lines) over the actual travel distance (the actual distance travelled by each subject reaching the respective number of targets).

Statistical Data Analysis

Data were analyzed using JMP 10.0 (SAS, Cary, NC) and used 2-tailed testing at α of .05. Parametric methods were used to analyze data that were normally distributed or could be transformed to a normal distribution, otherwise nonparametric methods were used. The primary, plus 4 secondary, game performance metrics were compared between the PC and AR versions of the game. Next, the relationship between game scores and motor status (BBT score) was calculated, with secondary analyses considering FM subscores. Finally, in an exploratory analysis, reaching times were compared according to the quadrant of visual space in which the fruit target first appeared, separately for the PC and AR game versions, and according to side of stroke.

Results

A total of 18 patients were recruited. All completed the study protocol, with no adverse events reported including no fatigue. Enrollees were on average 57 years old, 70 months poststroke, and had mild to moderate motor deficits (Table 1).

Table 1.

Subject Characteristics. ^a

n	18
Age	57 ± 14 years
Gender (female/male)	8/10
Dominant hand (right/left)	12/6
Stroke-affected side (right/left)	12/6
Time poststroke	70 ± 73 months
Hypertension	61%
Diabetes mellitus	22%
Atrial fibrillation	5%
FM total score	56 ± 11
FM hand/wrist subscore	21 ± 5
FM proximal subscore	30 ± 7
BBT score
Affected arm	38 ± 14
Affected/unaffected arm	0.66 ± 0.25

Abbreviations: FM, Fugl-Meyer; BBT, Box & Blocks Test.

a

Values are mean ± standard deviation or percentages.

Motor Performance With PC Versus AR Game Versions

By all measures, motor performance while playing the AR-based version of the Fruit Ninja game was superior to the PC-based version. Game score, the primary metric, averaged 58 ± 10 targets/game (mean ± SD; range 42-78) with the AR-based version, 21% higher (P = .0001) than when the same patients played the same game but using the PC-based version (48 ± 10 targets, range 34-71). All 4 secondary metrics were also significantly different between AR and PC versions and provided insight into these game scores. Reaching times with AR were 19% faster than with PC (870 ± 350 vs 1070 ± 510 ms, P = .0001); consistent with faster movements in the AR-based version, subjects traveled longer distances during 90 seconds of playing the AR-based version as compared to the PC-based version (22.113 ± 0.433 vs 19.998 ± 0.549 m, P = .0002). Movement time consistency was also better with AR, showing 15% less variability compared to PC (0.41 vs 0.48, P = .0068). Response latencies were 15% shorter with AR as compared to PC (1.71 ± 0.66 vs 2.01 ± 0.79 seconds, P = .0249). Movements were more efficient with AR as compared to PC (60.33 ± 14.8% vs 55.28 ± 14.4%, P = .0019).Visual inspection was consistent, indicating that movements during the AR-based game were more consistent and less erratic versus the PC-based game (Figure 2).

Figure 2.

Distribution of hand position for (A) all patients playing the augmented reality (AR) version of the game and (B) all patients playing the personal computer (PC) version of the Fruit Ninja game. The target locations and straight trajectories between targets are more prominent in the AR version, while movements are more inconsistent and erratic during the PC version, consistent with quantitative values for game metrics. Also pictured are the distribution of hand position for a single exemplary subject during (C) the AR version and (D) the PC version of the game.

The rate of task learning was similar between AR and PC based on improvement in scores over time. Analysis of the scores in consecutive rounds within each version shows that in both the AR and PC versions, performances improved significantly from round 1 to 2 (from 54.88 ± 9.24 to 60.28 ± 9.18 in AR, P = .0010; and from 45.11 ± 7.78 to 48.72 ± 9.98 in PC, P = .0257), but not from round 2 to 3 (from 60.28 ± 9.18 to 61.94 ± 9.66 in AR, P = .2795; and from 48.72 ± 9.98 to 50.22 ± 9.47 in PC, P = .2991). This suggests similar learning curves for the AR-based and the PC-based versions of the game.

Behavioral Correlates of PC and AR Scores

AR scores correlated significantly with all 3 motor assessments of the stroke-affected arm (Table 2 and Figure 3), particularly the BBT score. By contrast PC scores were related to motor status in only one instance and to a weaker degree as compared to AR scores. If a formal Bonferroni correction was applied to correct for multiple comparisons, only AR scores would remain significantly related to motor status.

Table 2.

Behavioral Correlates of AR and PC Game Scores.^a

Behavioral Measure	AR Game		PC Game
	r	P	r	P
FM hand/wrist subscore	.56	.0149	.36	.148
FM proximal subscore	.52	.0266	.32	.1957
BBT score, affected arm	.64	.0042	.48	.0433

Abbreviations: AR, augmented reality; PC, personal computer; FM, Fugl-Meyer; BBT, Box & Blocks Test.

a

Values in boldface represent significant relationships at P < .05.

Figure 3.

The top row shows correlation of augmented reality (AR) game scores with the proximal Fugl-Meyer (FM) subscores, distal FM subscores, and the Box & Blocks Test scores, from left to right; the bottom row shows correlation of personal computer (PC) game scores with the same.

Motor Performances in Relation to Quadrant of Visual Space

The quadrant into which each fruit target appeared during game play was classified in egocentric space, that is, either near or far from the body, plus either ipsilateral or contralateral to the stroke-affected arm. During game play with the AR setting, patients with right arm motor deficits (n = 12) showed the fastest reaching times when targets appeared in the near-contralateral quadrant, significantly faster versus targets appearing in any of the 3 other quadrants (P = .0001-.0002). The same was true when patients with left arm motor deficits (n = 6) played the game with the AR setting, with reaching times being fastest for targets appearing in the near-contralateral quadrant as compared to each of the other 3 quadrants (P = .0001-.0007). For both right arm and left arm groups, reaching times did not differ in relation to quadrant of space of target origin when the game was played using the PC setting.

Discussion

In the current study, patients with chronic poststroke hemiparesis used the affected arm to play the Fruit Ninja game using 2 different human–computer interfaces, one PC and one AR. During the AR-based version of the game, as compared to PC, scores were higher, movements were faster and more consistent, and performances were more tightly linked with arm motor function. These differences are likely due to the additional cognitive demands imposed when playing the PC-based game. The current findings underscore how choice of human–computer interface can influence task demands and thus behavioral performances and is thus likely to be important when designing a stroke rehabilitation protocol.

Although movement demands were identical across the 2 versions of the game, cognitive demands were not. Both versions of the Fruit Ninja game required subjects to make arm movements that rapidly integrated visual and somatosensory inputs.^25,26 During AR version play, subjects looked directly at their hand as it moved to control gameplay, but during PC version play, subjects instead looked at a computer screen, which increased cognitive demand. This modulation of cognitive demand can be understood in terms of a difference in the spatial congruence between movement goal and motor output, ²⁷ an uncoupling between eye and hand movements, ¹⁶ or the imposition of an added sensorimotor transformation. ²⁸ Regardless, the higher cognitive demand associated with playing the PC-based version of the game reduced game scores by 21% compared with the AR-based version (P = .0001), in association with slower and less consistent arm movements. Differences in cognitive requirements are particularly important after stroke, as demand increases having a negligible effect in healthy subjects can exceed cognitive reserve and degrade behavioral performance in subjects with stroke.^16,18-20

Although the increased cognitive demand in the PC-based game reduced movement quality (lower scores, slower reaching, less consistent movement), this might nonetheless be the preferred rehabilitation approach for some subjects such as those needing to strengthen visuospatial skills. The brain constructs a stable spatial representation of the workspace environment, ²⁹ transforming hand and target positions into a common body-centered space.^30-32 The AR game exists in this same space—spatial transforms for game play are the same as those used during real-world activities—but the PC game plays out on the computer monitor, which has its own coordinate space. As such, with a PC-based approach, the patient must perform at least one extra spatial transform to convert the virtual world’s coordinates into body-centered coordinates. A PC approach may thus be useful to promote cognitive recovery after stroke. Furthermore, increased cognitive demand modulates activity in brain motor networks and can enhance motor learning in healthy subjects and in subjects with stroke,^33-36 and so might have a role in some types of motor rehabilitation. Therefore, future studies might test the hypothesis that sustained practice of the PC game, as compared to the AR game, may be associated with a larger degree of motor improvement after stroke.

Advances in technology are providing many new types of computer interface to promote brain repair after stroke. The current study examined 2 forms, PC and AR, incorporating the popular Fruit Ninja game into each. A PC approach to gaming substitutes a nonimmersive virtual environment in place of the real world, ¹¹ shifting the subject’s attention toward the virtual world (computer monitor) rather than to the real world (the tabletop on which the paretic hand is moving). ³⁷ An AR approach to gaming overlays computer-generated virtual objects into the real world, thereby offering a direct interaction between user and computer output. Such real-world activities may be more intuitive than many PC-based approaches, which might promote massed practice. An AR approach might also allow patients to practice real-life functional tasks safely, for example, a patient might work on being independent in the kitchen by working with computer-generated objects that have zero risk of spilling, burning, or electric shock. It also provides standardization and consistency across repeated trials, features that are difficult to achieve in traditional therapy involving real objects; furthermore, adopting an AR approach allows tasks to be practiced in a structured way, using highly controlled environments, and can be gamified to encourage and intensify practice. Also, feedback (multimedia, error augmentation, etc) and task difficulty are readily adjustable to the level of disability and/or personal preferences of individual patients. A PC-based approach might nonetheless have advantages, for example, to promote recovery of visuospatial skills, or by using modulation of cognitive demands to promote other forms of recovery. The optimal choice of human–computer interface likely varies across individuals, over time, and according to specific goals. The current study provides insights into how choice of human–computer interface affects motor performance and thus might influence stroke rehabilitation and motor recovery.