Abstract
This study tested the use of a customized body-machine interface (BoMI) for enhancing functional capabilities in persons with cervical spinal cord injury (cSCI). The interface allows people with cSCI to operate external devices by reorganizing their residual movements. This was a proof-of-concept phase 0 interventional nonrandomized clinical trial. Eight cSCI participants wore a custom-made garment with motion sensors placed on the shoulders. Signals derived from the sensors controlled a computer cursor. A standard algorithm extracted the combinations of sensor signals that best captured each participant’s capacity for controlling a computer cursor. Participants practiced with the BoMI for 24 sessions over 12 weeks performing 3 tasks: reaching, typing, and game playing. Learning and performance were evaluated by the evolution of movement time, errors, smoothness, and performance metrics specific to each task. Through practice, participants were able to reduce the movement time and the distance from the target at the 1-second mark in the reaching task. They also made straighter and smoother movements while reaching to different targets. All participants became faster in the typing task and more skilled in game playing, as the pong hit rate increased significantly with practice. The results provide proof-of-concept for the customized BoMI as a means for people with absent or severely impaired hand movements to control assistive devices that otherwise would be manually operated.
Introduction
People with cervical spinal cord injuries (cSCI) constitute the majority (52.2%) of the 12 000 individuals who sustain a new spinal cord injury each year in the United States. 1 These individuals often use assistive devices to enable them to interact with the environment and compensate for their functional losses. In particular, for those with higher level cervical injuries, the control technologies of existing assistive devices are limited in both variability and functionality. Difficulties are exacerbated by the fact that these technologies—such as sip-and-puff control—are often nonintuitive and difficult to master, and they are not adaptable to the evolving state of the individual’s impairments.2,3 The majority of these systems operate as switching devices and do not provide users with natural continuous control.
Other challenges for people with cSCI include maintaining the strength, integrity, and use of remaining motor functions and possibly recovering the lost ones. 4 The residual body movements, if captured, can provide a means to operate external devices. Therefore, a BoMI can be a paradigm in which users learn to recognize and redirect their remaining motor functions to accomplish new tasks. This reorganization of motor coordination patterns has been widely used in controlling a prosthetic arm with EMG signals 5 and driving a wheelchair using tongue movements. 6
Devices that are currently available do not adequately address the important objectives of easy access, intuitive use, and support of continued motor recovery. The introduction of the present BoMI aims at providing a solution to these deficiencies. Our approach in this study is built on an initial BoMI design that was based on tracking active markers by infrared cameras.7,8 In the proposed BoMI system, a high dimensional signal space captured from the paralyzed user’s residual movements was mapped onto a lower dimensional space of control variables.
The array of degrees of freedom captured from the human body by a system of wearable motion sensors is a natural resource in motor learning. Importantly, these preserved degrees of freedom exceed the number required to perform tasks such as typing on a computer keyboard or driving a wheelchair. This “kinematic redundancy” provides the BoMI user with a unique opportunity to identify and coordinate a convenient subset of movements to achieve task objectives with a flexible and adaptable motor behavior. 9 In the current BoMI system, inertial measurement units (IMUs) are used to capture localized body movements. This enables the user to control a cursor on screen via reorganization of high dimensional upper-body movements.
Performance measures during practice of different functional tasks demonstrate that the BoMI can simulate motor learning and provide a new pathway to independence and partial functional recovery to people with cervical spinal cord injury. The purpose of the present investigation was to evaluate the utility of this novel BoMI controller system for operating external devices by individuals with chronic cSCI. We hypothesized that such an interface would affect accuracy, smoothness, and coordination of upper-body movements; such that with practice, participants will learn to effectively reorganize their remaining movement abilities and thereby become faster and more accurate in controlling a computer.
Methods
Apparatus
Each participant wore a custom-made vest with 4 IMUs (MTx by Xsens Technologies BV, Enschede, Netherlands) attached to it (Figure1A). With this setup, the IMUs captured scapular retraction, protraction, elevation, and depression. The orientation of each IMU was measured by a sensor-fusion algorithm through the combination of the output of 3-degree of freedom embedded accelerometers and gyroscopes. For the purpose of this study, we used roll and pitch angles as input signals for the interface.
Study setup (A). Participants sat in front of a computer monitor wearing 4 inertial measurement units on the shoulders. Signals from the sensors were mapped into the 2D position of a computer cursor used to perform the different tasks; Sample reaching movement from one participant (B) first session versus (C) last session. The circles illustrate the center and peripheral targets, and each line represents the cursor’s path during one center-out reach.
The IMU signals were captured in real-time at the rate of 50 Hz (MATLAB Simulink, Mathworks. Natick, MA). The “body space” was defined by an 8-dimensional vector of coordinates generated by the 4 sensors (h). Principal component analysis 10 was used to construct a map between the body movements and the 2-dimensional “task space,” the computer screen (p). During calibration, participants were asked to perform a “body dance,” consisting of 1 minute of free upper-body movements. The map between the body movements and the task space was then defined by the first 2 principal eigenvectors of the calibration data (A).
Due to the initial differences in residual abilities of the participants, immediately after the calibration, the cursor’s origin and gains were set in the task space during a customization phase. The gains associated with each principal direction were normalized by the standard deviation of the projected data along the corresponding axis. The components of the body-to-task space map were then scaled to ensure that every point in the task space could be comfortably reached. 7 Furthermore, the origin of the calibration data was offset to be the center of the task space (p0). This process ensured users’ ability to reach the entire task space. Small involuntary movements of the shoulders were filtered to prevent unwanted jitter of the cursor. 11
Interface Modules
The interface was developed to serve as a comprehensive motor training tool that promoted learning through useful and engaging tasks. The users were engaged in a variety of tasks, each involving different movement sequences and strategies to fulfill its specific objectives. This approach is supported by multiple studies,12,13 suggesting that the consolidation of learning and long-term retention of the acquired skills is achieved to a greater extent by practicing multiple diverse tasks rather than by focusing on a single task. Therefore, the tasks comprised the following 3 modules. For further technical details of the interface modules, see Farshchiansadegh et al. 11
Reaching
In order to investigate the motor learning, a center-out reaching task was implemented in the interface. The targeted reaching in 2D is a specific motor skill that has high relevance to a broad variety of control endeavors that require the accurate and timely positioning of a control point. Examples of such endeavors are the operation of powered wheelchairs and the interaction with personal computer systems.
Participants used the cursor to reach to 8 peripheral targets (1.1 cm radius) placed at equal intervals on an 11.4 cm diameter imaginary circle. Each target was presented 3 times in a randomized order. Participants moved back to a center target after the completion of each reach. The task was to acquire the target within 1 second and to hold the cursor inside—for 500 milliseconds—until the target disappeared. If the reaching time exceeded this limit, the color of the target changed from green to red, but the trial was complete only after the participant reached the target and remained inside its boundaries for 500 milliseconds.
Typing
An on-screen virtual QWERTY keyboard (http://cnt.lakefolks.com/) was adapted and implemented in the interface. The dwelling time to select a key for display was set at 300 milliseconds. Participants typed the English pangram sentence “The quick brown fox jumps over the lazy dog” during each practice session. This task required the user to stabilize the cursor over each key for a successful character display. Failure to stabilize the cursor resulted either in the inability to generate text or in the production of erroneous text (if the cursor rested over the wrong key), requiring corrective actions. Participants were instructed to correct for typing errors.
Gaming
To promote learning and retention and to make practice sessions entertaining, we created a library of flash video games such as solitaire, spades, and so on. Participants played a game of their choice for 10 minutes during each practice session. In order to be able to track the performance, we also developed a 2-dimensional virtual “Pong” game in MATLAB. They were required to repeatedly hit a virtual ball with a virtual paddle. The ball would bounce back upon hitting a wall. Therefore, the skill of the player could be measured in terms of paddle/ball hit-rates. Pong was expected to improve the temporal and spatial accuracy of the movement. Each block of the Pong lasted for 2 minutes.
Participants
Eight individuals with chronic cSCI (3 females; ages 29-58), on average 12.50 ± 7.67 years postinjury, with C4-C6 injury levels, participated in the study. Participants were excluded if their injury was below C6 or they had lower-body movements. Table 1 lists the detailed demographic information. All participants were naïve to the BoMI and provided informed consent approved by Northwestern University’s Institutional Review Board.
Participant Demographics and Injury Information.
Abbreviation: ASIA, American Spinal Injury Association.
Experimental Protocol
Each participant practiced for 24 sessions, twice weekly over 12 weeks. Each session lasted for approximately 1 hour. Participants had 1-week breaks after every 8 sessions. The calibration was done in the first session and all participants except one (S1) continued to use the same map during the remaining sessions. S1 asked for a recalibration for the first 3 sessions. Each session begun with a reaching task, followed by the typing, 5 blocks of the “Pong” game, 10 minutes of the game of choice, and ended with another reaching task.
Data Analysis
Position data were interpolated to ensure a fixed 50 Hz sampling rate and filtered using a sixth-order Savitzky-Golay smoothing function 14 with ~7 Hz cutoff frequency.
As an indicator of changes in skill levels in using the BoMI controller, we examined changes in the time required for individuals to reach the target positions using the controlled cursor. The onset of movement was detected as the cursor leaving the circumference of the origin target. Movement time was calculated from the onset of movement to the target disappearance. Movement error was calculated as the distance between the cursor and the target at 1 second after the onset of movement. Straightness of movement was captured by the “aspect ratio,” that is, the ratio of the maximum deviation from the straight line connecting the beginning and end of movement to the minimum distance between the two. A value of zero for the aspect ratio indicates the straight line movement. Finally, smoothness of movement was quantified by the first temporal derivative of hand acceleration, or “jerk,” whose value is smaller for smoother movements. 15 Here, the normalized dimensionless jerk was used, defined as the square root of the integrated jerk based on Teulings et al. 16 These metrics are widely adopted to evaluate cursor-control performance in brain machine interfaces. 17 Outlier trials (7.8%) were determined by applying the outlier labeling rule with a factor of 2.2 to the movement time vectors. 18
Additionally, pong blocks were evaluated by counting the number of hits per minute. Typing data were evaluated as the number of characters typed per minute. Participants started practicing typing and pong modules in the third session. Therefore, typing and pong hit-rates were calculated in session 3 versus session 24.
Statistical Analysis
A repeated-measures mixed model was used to evaluate if outcome measures improved from session 1 to session 24 accounting for correlation of repeated measurements from each participant. The data were fit to a linear mixed model with Session and Target as the effects. For the reaching performance measures (movement time, movement error at 1 second after the onset of movement, aspect ratio, and normalized jerk), the data were log transformed to ensure that the residuals of the model were normally distributed. An analysis of variance was then performed on the linear mixed model to evaluate the effect of session on each of the performance measures. Means and standard deviations reported are the nontransformed value, while the F and P statistics reflect the log transformed data.
A paired t test was used to evaluate the performance improvements for the pong game and typing task from session 3 to session 24. The hit-rates from 5 blocks of the pong game were averaged for each participant to account for the missing blocks. Hence, the result was one measurement per session for each participant. Also, participants performed only one typing task per session.
All statistical tests were 2-sided at the 5% significance level. Statistical analyses were performed with R version 3.2.3 (www.R-project.org).19,20
Results
The evolution of the kinematics of the reaching movements from a typical participant is shown in Figure 1B and C. The results from repeated measures mixed model showed significant movement time reduction over the 24 sessions of practice (session 1 time = 7.12 ± 5.78 seconds vs session 24 time = 2.85 ± 1.51 seconds, F1,7 = 39.44, P < .001; Table 2).
Performance Evaluation Metrics a (shaded cells are for missing data).
Data are presented as mean ± standard deviation.
Value is smaller for straighter movements.
Unit: ×103.
All participants increased their accuracy by managing to get significantly closer to the targets at 2 second after the onset of movement (session 1 movement error = 6.09 ± 10.82 cm vs session 24 movement error = 1.75 ± 1.77 cm, F1,7 = 48.91, P < .001; Table 2). In addition to the task-specific performance improvements, as a result of practice, all participants straightened out the reaching paths (session 1 aspect ratio = 0.71 ± 0.99 vs session 24 aspect ratio = 0.24 ± 0.18, F1,7 = 44.34, P < .001; Table 2) and made smoother movements to targets (session 1 normalized jerk = 8.81 ± 17.57 vs session 24 normalized jerk = 0.89 ± 1.47, F1,7 = 35.13, P < .001; Table 2). The learning curves of the performance measures are shown in Figure 2.
Learning curves for reaching movement measures throughout the 24 sessions of practice; the black dots are individual session averages across all participants and the error bars are the 95% confidence intervals.
The movement path of the paddle and the ball hits from a typical subject is shown in Figure 3. We looked at the average hit rate in session 3 versus session 24. Since S1 started playing pong in session 4, this participant was excluded from analysis. A paired t test performed on the remaining 7 participants showed everybody hit the moving ball more times toward the end of practice (session 3 hit rate = 5.26 ± 2.15 min−1 vs session 24 hit rate = 19.59 ± 9.57 min−1, t = 4.24, P = .006; Table 2).
Increase in pong hit rate showed by sample pong trajectories from the very first 2-minute pong block in session 3 (top) compared to the very last 2-minute pong block from session 24 (bottom) played by a typical participant; (A) and (C) show the vertical paddle trajectory (thick line), ball vertical trajectory (dotted line), and the hits (dots) versus time; (B) and (D) show the paddle paths (lines) and the location of hits (dots); the rectangle marks the game area.
The overlaid typing trajectory of a sample participant on the virtual keyboard is shown in Figure 4. Since S2 started typing later in her practice sessions, she was excluded from analysis. A paired t test performed on the remaining 7 participants revealed a significant increase of typing rate across practice (session 3 typing rate = 8.56 ± 4.64 characters/minute vs session 24 typing rate = 14.67 ± 2.62 characters/minute, t = 3.44, P = .14; Table 2). The maximum typing rate was achieved by S8 and was 18.82 characters per minute.
Increase in typing rate displayed by sample typing trajectories from the very first typing session (top) compared to the very last typing session (bottom) for a typical participant; (A) and (C) show the vertical (black line) and horizontal (grey line) cursor movements and the typed characters (dots); (B) and (D) show the same information in the first minute.
Discussion
The present study demonstrated that people with high-level spinal cord injury could effectively operate assistive devices by employing functional reorganization of their residual upper-body movement skills.
There are several technological solutions for providing targeted assistance to people with severe paralysis. In particular, creative solutions are available for controlling powered wheelchairs, such as sip-and-puff systems and various configurations of head switches. 21 A recent clinical study has investigated control of powered wheelchairs using movements of the tongue. 6 While all of these are effective approaches, they do not address 3 key objectives: (a) providing a flexible interface that can adapt to the evolving skills and conditions of the users, (b) promoting the support and engagement of the user’s full residual mobility, and (c) providing a continuous and proportional control interface rather than a discrete on/off control. These objectives guide the development of the proposed BoMI concept.
The BoMI using residual body movements not only facilitates the control of an external device but also engages the users in a new goal-directed and sustained physical form of activity that has the potential to prevent muscle atrophy and to maintain mobility. 22 This is especially valuable for individuals with higher-level SCI, among whom exercise of residual movements often occurs less frequently than desired. 23 An important goal in developing an intelligent body-machine interface is to offer a medium through which the users can recover 3 key features of unimpaired motor control:
Accuracy, that is, the ability to reach a target—which could be a desired velocity of a wheelchair or a key on a keyboard—with sufficient confidence and reliability. 24
Smoothness, that is, the ability to generate continuous motions without unintended jumps or oscillations. 15
Coordination, that is, the ability to engage in well-crafted temporal patterns of movements. 9
In order to assess the effectiveness of our BoMI in enabling these features, multiple parameters were extracted from the center-out reaching task. As a result of practice, participants learned to reach the targets in shorter times with higher accuracy. Moreover, it is evident that during the learning process, reaching trajectories became straighter and smoother. These findings indicate that participants grasped the control of the interface and were able to improve their performance over time. Although the average movement time of 2.58 seconds achieved by participants in session 24 is slightly worse than the movement time of 1.1 seconds reported in a brain-computer interface study, 25 the study had only 2 participants with the number of reaching blocks 544 and 200 compared to 48 in this study.
Improvements in motor control were mirrored by similar improvements in functional tasks. For “pong” and typing task the participants achieved a higher level of spatiotemporal performance. In the pong task, participants significantly improved the hit-rate across practice, while in typing, they achieved significantly higher characters per minute. It should be noted that although the participants were not encouraged to type as fast as possible, the highest typing rate achieved using the BoMI (18.8 characters/minute; average across all participants 14.7 ± 2.6 characters/minute) was higher than the highest typing rate previously reported in brain-computer interface systems, which is 15.4 characters/minute. 26
Our results provide a proof of concept for the use of the proposed BoMI as a new and potentially effective approach for enabling and enhancing the recovery of independence after cervical spinal cord injury through the control of devices by residual body movements. This new paradigm, unlike the currently commercially available technologies, provides the user with a continuous, proportional, and more intuitive controller. Using the “body” as one end of the BoMI helps engaging the user’s residual body movements in functional and entertaining tasks. Working with the interface can be a novel approach to combine both the assistive and the rehabilitative goals within a single framework. The design of this study lacked a control group with unimpaired users and the long-term follow-up to investigate long-lasting improvement in performance. However, the main purpose of the study was only to provide a proof of concept on the ability of this interface to detect relatively small movements that do not require significant effort by the user. Future studies will focus on the use of the proposed BoMI as a rehabilitative medium, long-term effects, and user satisfaction with the interface.
Footnotes
Acknowledgements
The authors are grateful to Ms Laura Pickering and Ms Emily Lazzaro, the physical therapists on the team, for their help with the evaluations, and to Dr. Julia Jungwa Lee for statistical advice.
Declaration of Conflicting Interests
The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This study was funded by US National Institute of Child Health and Human Development, Grant 5R01HD072080 and NIDRR grant H133E120010.