Abstract
Background. Adaptive mixed reality rehabilitation (AMRR) is a novel integration of motion capture technology and high-level media computing that provides precise kinematic measurements and engaging multimodal feedback for self-assessment during a therapeutic task. Objective. We describe the first proof-of-concept study to compare outcomes of AMRR and traditional upper-extremity physical therapy. Methods. Two groups of participants with chronic stroke received either a month of AMRR therapy (n = 11) or matched dosing of traditional repetitive task therapy (n = 10). Participants were right handed, between 35 and 85 years old, and could independently reach to and at least partially grasp an object in front of them. Upper-extremity clinical scale scores and kinematic performances were measured before and after treatment. Results. Both groups showed increased function after therapy, demonstrated by statistically significant improvements in Wolf Motor Function Test and upper-extremity Fugl-Meyer Assessment (FMA) scores, with the traditional therapy group improving significantly more on the FMA. However, only participants who received AMRR therapy showed a consistent improvement in kinematic measurements, both for the trained task of reaching to grasp a cone and the untrained task of reaching to push a lighted button. Conclusions. AMRR may be useful in improving both functionality and the kinematics of reaching. Further study is needed to determine if AMRR therapy induces long-term changes in movement quality that foster better functional recovery.
Keywords
Introduction
Functional recovery is possible for people with chronic stroke,1-3 but the best way to administer rehabilitation to achieve this recovery is not well understood. Specific challenges to providing therapy after stroke for the upper extremity include addressing highly varied combinations of impairments, motivating the participant to perform repetitive therapy, promoting active learning, and balancing recovery of lost function and motor patterns.4-6 The adaptive mixed reality rehabilitation (AMRR) system presented here begins to address these challenges using an interactive framework to train motor components related to both the completion of an activity and the quality of the movement during the task. AMRR integrates repetitive task training using smart objects with real-time motion capture and analysis to extract kinematic measurements as a useful quantification of arm motor performance 7 and provide a systematic assessment of typical upper-extremity impairments. The kinematic data are also used to provide real-time and summary audiovisual feedback to the participant for self-assessment of the movement. The interactions are engaging to motivate task completion and promote generalized learning of motor elements related to the task. The tasks and feedback may be adapted to focus on activity completion or on performing the activity within the additional context of reestablishing premorbid movement patterns (ie, by improving movement quality) based on the participant’s therapy needs. Feedback is presented within the context of a functional task, and the underlying principles for transitioning to different tasks or feedback streams are consistently applied to all participants. The design principles of the system have been presented elsewhere.5,6
Other researchers have developed virtual environments to provide feedback on therapeutic tasks,8-10 which have generally produced positive outcomes in people with stroke. Whereas feedback can enhance motor learning, 11 the extent to which interactive therapies differ from traditional therapy is still under investigation. The purpose of the present study was to gather fundamental information on the value-added potential of the AMRR system by examining functional and kinematic outcomes from 2 groups of participants with chronic stroke who received either traditional repetitive task reaching therapy (performance feedback provided by the therapist) or AMRR reaching therapy. Although we anticipated that both groups would benefit from receiving a month of physical therapy, we hypothesized that the AMRR therapy group would show greater and more generalized improvements in overall movement quality (eg, more efficient trajectories, smoother acceleration and deceleration, and reduced compensatory movements) because of active learning of associated motor elements. Our kinematic evaluation paradigm also aims to address an evasive and difficult issue—the quantification of quality of limb movement—that represents a substantial step toward discovering the relationship between the recovery of motor performance and the recovery of function.
Methods
Participants
A total of 25 participants with stroke were enrolled from Banner Baywood Medical Center and provided informed consent. The protocol was approved by the Banner Health and Arizona State University institutional review boards. Participants were placed into the AMRR group (n = 11) or the control group (n = 10), based on random group assignments for the first 4 participants and subsequently adaptive randomization in an effort to minimize group differences in impairment severity, age, and time poststroke. Inclusion criteria were the following: age between 35 and 85 years old, currently 6 months to 5 years poststroke, right-sided hemiparesis, and right-hand dominant prestroke. Active range-of-motion (ROM) criteria (adopted from the EXCITE trial 12 ) were the following: shoulder flexion of at least 45°, elbow ROM of at least 30° to 90°, forearm rotation of at least 20° pronation or supination, wrist extension of at least 20°, and at least 10° active extension of the metacarpophalangeal and the interphalangeal joint of the thumb and any 2 fingers. Exclusion criteria were current or past seizure disorder; change in pain medication or alcohol use immediately prior to the study; current video game playing exceeding 1 h/wk; concurrent participation in another physical therapy program; injections of antispasticity drugs within the past 3 months; inability to distinguish depth, shape, color, motion, pitch, and timbre as measured by a sensory perception test; and a score ≤24 on the Mini-Mental State Examination. Table 1 shows participant demographics as well as severity, baseline total upper-extremity Fugl-Meyer Assessment (FMA) score (maximum score of 126), and lesion location. The treating therapist and attending rehabilitation doctor determined the severity levels using the following: estimation of muscle tone, strength (mild, 4− to 5; moderate, 3− to 3+; and severe, less than or equal to 2+, as measured by the Manual Muscle Test), active ROM (mild, within functional limits; moderate, significantly greater than inclusion criteria but less than functional limits; severe, meets inclusion criteria), and subjective assessment of the participant’s coordination during finger-to-nose and reaching movements.
Demographic Information and Baseline FMA Score (out of 126) for the AMRR (MR) Participants and the Control (C) Participants a
Abbreviations: FMA, Fugl-Meyer Assessment; AMRR, adaptive mixed reality rehabilitation; M, male; F, female; L, left; ICH, intracerebral hemorrhage.
Group summary or median values and (1st, 3rd) quartiles shown at the bottom where appropriate.
Evaluations
Each participant completed an evaluation, consisting of validated clinical tests and reaching tasks, within 5 days before starting and within 5 days after completing treatment.
Clinical outcome measures
The clinical scales included the following: the Motor Activity Log (MAL), 13 a validated scale (0, not used, to 5, used as much or as well as before the stroke) that allows stroke survivors to self-report their amount of use and quality of movement of their more impaired arm during activities of daily living; the full Stroke Impact Scale (SIS), 14 a normalized, validated scale (0%, no recovery, to 100%, full recovery) to measure the self-reported impact stroke has had on areas such as social interaction, emotion, motor function, and cognition; the Wolf Motor Function Test (WMFT), 15 a therapist-administered validated scale that rates (functional ability score of 0, could not perform, to 5, normal movement) and times complete upper-extremity movements related to functional tasks; and the upper-extremity FMA,16,17 a validated scale that includes joint ROM, pain, sensation, and proprioception (0, no function, to 60, full function in all areas) and motor function of the affected arm (0, no function, to 66, full function in all areas), for a total possible score of 126. The evaluating therapist who performed the FMA and WMFT was blinded to the treatment group designation. The researcher who administered the SIS and MAL was not blinded to the treatment group.
Reaching tasks
The participants completed a trained and an untrained reaching task during which motion capture data from the right hand, right arm, and torso were collected. The trained task was 4 sets of 10 reach-to-grasp movements to 4 stable cone objects. The 4 locations used were at the midline with the cone on the table (gravity-eliminated movement) and 6 inches off the table (antigravity movement) and on the right side with the cone on the table and 6 inches off the table. The target distance was based on the individual participant’s active-assisted reach. This location was used for both the preevaluation and postevaluation as well as the therapy and gave the participant a therapy goal beyond his or her baseline active reach. All reaches started from a consistent rest position (marked with tape and monitored by the therapist) and were self-initiated at the participant’s own pace. Reaching to grasp a cone was a therapy task available to both groups, although the time spent on that task varied per person based on the therapist’s discretion and was more likely to be used during AMRR sessions.
The untrained movement task was a series of reaches- to-touch 1 of 9 buttons (1 inch) embedded within a rectangular upright stand (button box test). 18 The 9 buttons were arranged in 3 rows and 3 columns, equally spaced by 5 inches, placed so that the middle button aligned to the participant’s sternal angle. Each button lit 3 times in a pseudorandom order. The light provided an external stimulus to initiate the reach, and the light turned off when the button was successfully touched. The participant was instructed to reach the target as quickly as possible. The location of the target button was unknown to the participant until immediately prior to the start of the reach. The elements of this task that differentiate it from the trained tasks are unknown target location, instruction to move as quickly as possible once the button lights, triggering from an external stimulus within a defined latency, smaller targets, and greater off-sagittal target locations.
Treatment Protocols
Participants from the AMRR group received 1 hour of therapy (not to exceed 120 reaches) 3 times a week for 4 consecutive weeks (12 sessions in total). Participants from the control group received therapy of a matched amount of time on the same schedule. Tasks in both groups were performed with the right hand only. The same therapist administered treatment to both groups. The control group received upper-extremity therapy, exemplified by pegboard reaching tasks, bead threading reaching tasks, cone reaching tasks, and ROM and coordination exercises. During therapy with more severely impaired participants, the therapist focused on real-time instruction based on her perception of movement quality, whereas she tended to summarize feedback to higher functioning participants after they finished a set of reaches. If the participant was not improving with verbal cues, the therapist guided the arm with active-assisted movements for a few reaches. The therapist also monitored compensation by using her hand to provide a tactile cue related to excessive shoulder or torso movement.
The AMRR group performed reaching tasks to 3 different objects: a virtual point (no physical target), a 3-inch physical button, or a physical cone. The targets were either on the table or 6 inches above it. Motion capture data collected from the participant’s right hand, right arm, and torso were transformed into kinematic measurements (see description of kinematic assessments below) and used to provide audio and visual feedback to the participant during the task, evaluate the progress of the participant, and adapt the therapy accordingly. The feedback provided real-time visual cues about trajectory error and hand rotation and real-time audio indications of the speed of the hand’s movement, elbow extension, and torso and shoulder compensation. The system also provided audio and visual cues indicating the task was successfully completed and provided a visual summary of where the trajectory errors had occurred to aid in forward planning. A picture of a participant using the system is shown in Figure 1. Further explanation of the feedback theories and applications4,5 and detailed descriptions of 2 participants’ adaptive journeys through AMRR therapy 19 have been presented elsewhere.

Participant (center) using the adaptive mixed reality rehabilitation system with the treating therapist.
Data Analysis From Pre-Post Evaluations
Kinematic assessments
The kinematic parameters calculated during the trained task (cone reaches) and untrained task (button box) cover both activity level and body function aspects of recovery 6 and are representative of key areas of impairments caused by a stroke.5,20 The parameters for the trained task kinematic analysis were grouped by .common functionality into the following 7 categories: (1) trajectory—horizontal and vertical trajectory efficiency and consistency; (2) targeting—error and consistency of hand placement during the manipulation phase of the reach; (3) temporal—peak reach speed magnitude and consistency and time to target; (4) velocity—smoothness of the reach and how well the velocity curve adheres to an ideal bell shape; (5) compensation—excessive shoulder or torso movements during the reach; (6) joint function—that is, shoulder, elbow, and wrist ROM and consistency; (7) joint correlation—synergy between key joint pairings of the arm. 5 The kinematic parameters were further normalized into a Kinematic Impairment Measure (KIM) score. 20 The KIM maps each parameter onto a distribution function that ranges from 0 (within the idealized range of movement) to 1 (maximal deviation from the idealized range), modeled on a set of reaching data from both able-bodied people and people with stroke at different levels of impairments (ie, mild, moderate, and severe). Table 2 illustrates how ranges of KIM values map to raw data for 3 example kinematic attributes: peak hand movement speed, maximum torso flexion during the task, and the correlation value between 2 joints. The conversion of data from attribute-specific units of measurement to a common scale allows for parameters to be combined into the 7 category level KIMs and for category values to be combined as a weighted average into 1 composite KIM, providing higher level summaries of performance. The normalization also reorients all parameters to have the same directionality of improvement. For instance, peak speed (second column, Table 2) of the hand is considered unimpaired at between 0.42 and 0.60 m/s. When the speed either increases (eg, as a result of ballistic movement) or drops (eg, as a result of weakness), indicating an elevated impairment, the KIM correspondingly increases. KIM values were calculated for each location of the cone reaches and each button location and, respectively, averaged together to obtain the kinematic assessment for the trained and untrained tasks. The percentage change in composite KIM for the trained task after treatment—(Composite KIM post − Composite KIM pre)/(Composite KIM pre × 100%) and transformed so a positive change indicates a reduction in KIM—was calculated for each participant. This percentage change gives a normalized measure of the overall change in movement quality for each participant. A subset of 4 parameters, chosen from the parameters comprising the 7 categories, was used to evaluate the untrained task. These categories (2 activity levels —trajectory and velocity—and 2 body functions—torso compensation and joint correlation) were chosen to represent the data because they are most relevant to the button reach task. Composite and category KIM for the untrained task were not compared because the ideal weights for combining the parameters are still under investigation.
Mapping Between Raw Value Ranges of 3 Example Kinematic Attributes and the Kinematic Impairment Measure (KIM) Values Associated With Those Ranges
Statistical Analysis
Because of small sample sizes, all statistical analyses were performed using nonparametric methods. The demographic data for the 2 groups were compared using the 2-sample Wilcoxon ranked sums test to demonstrate equivalent participant groups. Data from the clinical scales were analyzed within each group using a paired Wilcoxon signed-ranks test and between groups using the 2-sample Wilcoxon ranked sums test. Demographic and clinical scale data were tested for significance with α = .05. The overall composite KIMs and each of the 7 category KIMs (see previous subsection for details) derived from the trained task reach kinematic data were assessed within groups using paired Wilcoxon signed-ranks tests to determine group changes after treatment. Each comparison used a significance level of α = .01875, calculated by applying a Bonferroni correction for the 8 comparisons to a maximum family-wise error of .15. The distributions of the percentage change in composite KIM were compared between groups using a Kolmogorov-Smirnov 2-sample test, which is used to determine if 2 distributions are identical in terms of location (mean/median) and variance at α = .05. The 4 kinematic parameters (see previous subsection for details) derived from the untrained task kinematic data were also assessed within groups using paired Wilcoxon signed-ranks tests to determine changes after treatment. Each comparison used a significance level of α = .0375, calculated by applying a Bonferroni correction for the 4 comparisons to a maximum family-wise error of .15. The application of conservative Bonferroni corrections to an increased family-wise α of .15 was an attempt to balance the occurrence of type 1 errors (false positives) and loss of power 21 in analyses with multiple comparisons.
Results
In all, 4 participants (2 from the AMRR group and 2 before group assignment) withdrew from the study for unrelated medical reasons or because of unreliable transportation. The resulting 2 groups that completed treatment were not significantly different in age, time poststroke, or baseline FMA (all P values >.05, data shown in Table 1).
Clinical Scale Results
Median values and first and third quartiles (shown in parentheses) for the WMFT, FMA, MAL, and SIS are shown in Table 3 with significant P values bolded. Both the AMRR group and control group significantly improved their WMFT FAS score. Neither group had a significant change in their median time to completion of the WMFT tasks. Both groups increased their FMA Motor Function scores significantly, but only the control group significantly increased the scores in the ROM/pain/sensation sections. Neither group had a significant increase in the MAL for either the amount of use or quality of movement sections. The control group significantly improved their SIS score from 64.41% recovery to 76.48%. Between-group analysis of the FMA scores showed that the control group’s increase was significantly greater than that of the AMRR group in both sections. No other between-group analyses of the clinical scores was significantly different.
Clinical Scale Data (Median Values and First and Third Quartiles) Measured Pretreatment and Posttreatment a
Within-group comparisons are shown immediately after the group data, and between-group comparisons are shown in the last column. Significant P values are bolded.
Kinematic Results: Trained Task (Cone Reaches)
The median values (filled circle) and first (bar below the median) and third (bar above the median) quartiles for the composite KIMs, histograms of the percentage change of the composite KIMs of all participants, and category KIMs for each group for both groups are shown in Figures 2A, 2B, and 2C, respectively. The AMRR group showed a significant reduction (P < .001) in composite KIM (indicative of an improvement in overall movement quality), whereas the control group’s improvement was not statistically significant (P = .065). The distributions of the percentage change in composite KIM were characterized by mean improvements of 50.76% (±13.59%) and 15.69% (±54.47%) for the AMRR and control groups, respectively. The 2 distributions were significantly different (P = .023). The AMRR group showed a significant reduction in KIM for 6 of 7 kinematic categories (P < .001 for velocity, compensation, joint function, and joint correlation categories, and P < .005 for trajectory and temporal profile categories). Targeting was unchanged. The control group showed a significant reduction in only joint correlation KIM (P < .01).

Kinematic impairment measure (KIM) scores for the trained cone reach task performed during the preintervention and postintervention evaluation. The composite KIM values (A) and category KIM values (C) for the adaptive mixed reality rehabilitation (AMRR) group (solid dark line) and for the control group (light dashed line) are shown as median values (filled dot) and the first and third quartiles (upper- and lower-horizontal bars). Figure 2B shows histograms for the participants’ percentage changes in composite KIM score following treatment. The AMRR group’s distribution is shown with dark, solid bars and the control group’s is shown with light, striped bars.
Kinematic Results: Untrained Task (Button Box Reaches)
The kinematic results from the button box reaches were analyzed similarly to the cone reaches and are shown in Figure 3. The AMRR group improved in trajectory (P < .001), torso compensation and joint correlation (both with P < .005). The control group significantly improved in only trajectory (P = .027).

Kinematic impairment measure (KIM) scores for the untrained button reach task performed during the preintervention and postintervention evaluations. The category KIM values for the adaptive mixed reality rehabilitation (AMRR) group (solid dark line) and for the control group (light dashed line) are shown as median values (filled dot) and the first and third quartiles (upper- and lower-horizontal bar).
Discussion
This study examined kinematic and clinical outcomes of an AMRR system compared with traditional reaching therapy. AMRR participants showed statistically significant improvements in 6 of 7 kinematic categories (excluding targeting) that are representative of upper-limb impairments seen after stroke. The targeting category uses end-point error during the grasping phase as a metric of accuracy, adapted from previous studies.22,23 However, those studies investigated a reach-to-touch or reach-to-point task to a small target, whereas studies that explored reach-to-grasp movements did not use end-point error as a measure of grasp accuracy.24,25 A successful grasp of the cone object required consistent hand placement, resulting in both groups having low initial targeting KIMs and very little room for improvement.
The composite KIM (weighted average of category KIMs) results show that AMRR therapy can induce highly consistent improvements in movement quality, even though each participant received a unique training regimen. Previous studies have questioned how factors such as sex, 26 age, 27 lesion location, 28 impairment severity,29,30 and time poststroke25,31 affect motor recovery; however, the low variance among participants’ change in KIM scores after treatment suggests that AMRR therapy would be effective for participants with different demographic and clinical characteristics. Further data must be acquired to definitively state that the positive outcomes related to AMRR therapy are sufficiently robust to apply to participants with variable attributes.
Kinematic improvements were also seen in the untrained button reaching task, indicating that motor strategies learned during AMRR therapy were applied to a movement condition in which the participant can anticipate neither the initiation nor the specific direction of movement, both of which require different motor and cognitive abilities than were trained.32,33 Traditional therapy did little to induce favorable changes in overall kinematic-based impairment measures, neither in the trained task (Figure 2) nor in the untrained task (Figure 3). A possible explanation for this failure to have an impact on KIM scores is that traditional therapy often focuses on regaining function and less on reducing compensatory movements or recovering premorbid movement patterns. Even if a therapist wished to provide information on certain motor elements, assessing and providing feedback on multiple movement aspects in synchrony with the movement would be impossible. Although the exact relationship between increased movement quality and improved long-term recovery is still unknown, many researchers and clinicians speculate that addressing issues of movement quality and compensation is essential to sustaining and enhancing functional gains made during therapy.6,34,35
Both the AMRR group and the control group demonstrated similar statistically significant improvements in the WMFT, likely because both groups received the same dosage of therapy, and both focused on repetitive training of varying tasks and target locations. Both groups also had significant increases in the motor function portion of the FMA, with the control group demonstrating a larger change in that section. The control group also improved more in sections rating sensation, pain, and passive joint function. Control therapy participants had more flexibility in the target locations of their reaches to use a greater combination and magnitude of arm joint movements and may have also received manual therapy to the arm (such as ROM exercises, active-assisted reaches, or stretching). During the AMRR intervention, reaches were restricted to 4 designated locations in order to accurately calculate the KIM values and provide feedback, thus minimizing alternative directions of movement or therapist manual guidance. Future AMRR systems could include an increased number of target locations or could be used in conjunction with traditional therapy methods to address sensory compromise within the arm.
The control group also had a preferentially greater improvement in SIS score. Because the AMRR group was challenged to continuously improve their kinematics, the focus may have inadvertently shifted from how well they had done to how much more they could improve, resulting in lower perceptions of health-related quality of life. The AMRR system would likely benefit from a graphical interface that reflects therapy progress and provides motivation to continue making improvements. Neither group had significant increases in either section of the MAL, possibly because the MAL reporting time (previous 2 weeks) overlapped too much with the intervention time (previous 4 weeks) at the time of the postevaluation. Alternatively, this result could reinforce current evidence that arm performance during laboratory measurements is not necessarily predictive of use and functionality at home.36-39 To help participants better transfer the skills learned in the clinic toward activities of daily living, we are also creating a home-based therapy system 40 that will continue to reinforce the recovery of premorbid movement patterns as well as offering a wider variety of functional tasks to practice.
Limitations
Although the AMRR system appears to be a promising tool to deliver therapy focused on function and movement quality, the study presented had some limitations. This study used the same therapist to administer both therapy types, so the group results would not be skewed by the skills of the therapist. Future studies will include several therapists trained to administer both types of therapy and have them randomly assigned to participants. Further study is also needed on a larger, randomized sample, which will allow for a better generalization of the results and correlation analysis between the participants’ clinical profiles and their responses to therapy. The study also recruited an older population as well as people with lesions at any location resulting in right-side hemiparesis or multiple strokes. These liberal criteria may have confounded kinematic or clinical improvements as a result of preexisting joint, neurological, or movement impairments. The study also lacked a follow-up evaluation, so although short-term movement quality improvements were made in the AMRR group, the extent to which changes persisted after treatment ended is unknown. Finally, although the KIM scores have been positively correlated with validated clinical scores, 20 the relationship between movement quality and long-term recovery is still under investigation, making it difficult to truly determine how much improvement in movement quality is needed to become relevant to a person’s recovery after stroke.
Conclusion
The AMRR system provides a unique method of monitoring and addressing movement quality by providing integrated and adaptive feedback based on high-resolution motion capture during a therapy task. The AMRR therapy generated improvements in validated clinical scales and in kinematic measures during trained and untrained tasks. Although control therapy of matched dosage resulted in greater gains in the FMA, it failed to produce significant improvements in most of the kinematic parameters in any task. AMRR therapy appears to be an effective way to improve both function and movement quality, in an effort to enhance and sustain overall recovery.
Footnotes
Acknowledgements
The authors would like to thank Peter Nordland, Tina Schaffner, Barbara Lambeth, and Young An from Banner Baywood Medical Center for their clinical contributions to the project. We would also like to thank the students and faculty at Arizona State University who contributed to developing and administering the AMRR system.
Declaration of Conflicting Interests
The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The authors financial support for the research, authorship, and/or publication of this article declared receipt of the following: This article received financial support from the State of Arizona, the Banner Foundation and the NSF through CISE RI grant #0403428 and IGERT grant #0504647.
