Sage Journals: Discover world-class research

Abstract

An extremity rehabilitation program was proposed based on inertial measurement units (IMU) and virtual reality. A single IMU consists of a three-axis accelerometer, gyroscope, and geomagnetic sensors. One IMU is attached to the upper arm (master) and another to the forearm (slave). The IMUs are connected using a distributed sensor network implemented with interintegrated circuit communication. The motion-tracking algorithm running on a PC tracks the subject's hand based on the estimated IMU orientation and segment lengths through forward kinematics. The training contents, including various dynamic movements and static holds, were designed to evaluate the spatiotemporal aspects of the subject's functionality. The system was tested on a group of healthy subjects and a group with a simulated stiff elbow, allowing the evaluations to be quantitatively differentiated. The stiff elbow was simulated by taping the elbow to restrict the range of elbow motion. We expect the patients to be able to assess their own status without assistance from a therapist and select appropriate training methods to increase their rehabilitation effectiveness. Future studies will verify the availability and reliability of the upper extremity rehabilitation program for patients with a hemiplegia, leading to the development of an upper extremity rehabilitation program for three-dimensional movements of the upper extremities.

1. Introduction

A stroke, which is a type of cerebrovascular condition, is the third-highest cause of death in the United States [1]. Although stroke patients often regain consciousness after onset, 30–40% of patients suffer from hemiplegic complications such as a speech disorder or dementia, impeding their ability to live a normal life. Among the various disorders caused by a stroke, hemiplegia is very typical, with more than 80% of stroke patients displaying some form of hemiplegic disability [2]. The restoration of the upper extremity functionality is slow compared to the recovery of other functions such as posture or gait. Furthermore, if the rehabilitation is discontinued, patients may be unable to regain their normal upper extremity ability [1]. This poses a problem when cost and/or space limitations reduce the amount of rehabilitation training that patients can receive at a hospital. After discharge, patients must undertake rehabilitation at home [3]. Without the direction of a physical therapist, such patients may be unable to determine their own status or select an appropriate rehabilitation program.

In addition, traditional devices for upper extremity rehabilitation are simple and tedious to use, and patients may lose interest in them over the long term. If a patient does not engage in rehabilitation for the recommended time period, the efficiency of the process may decrease. To overcome this limitation, many studies have focused on a variety of training, confirming the results through a virtual reality system. Several studies using commercial gaming devices such as the Nintendo Wii or Microsoft's XBOX360 Kinect sensor have shown a clinical effect [4–6]. However, these systems do not evaluate the patient's functionality during training, which is necessary for patients to conduct rehabilitation at home on their own. Zhang et al. [7] attached an inertial sensor to the wrist and elbow joints of different patients and provided therapists with tools to remotely observe the rehabilitation movements of their patients in real time. The therapists were then able to design an appropriate exercise program according to the training progress of their patients. This system includes only three types of actions, that is, arm stretching, bending, and drinking water. Similarly, Willmann et al. [1] used a system that tracks the behavior of the upper extremity using inertial sensors. A physical therapist confirms the condition of the patient at a hospital directly or indirectly based on an examination of the stored data and then presents the next movement that the patient should practice. However, this system does not provide a systematic exercise method because it is dependent on the feedback of the physical therapist.

We previously presented an upper extremity rehabilitation system based on commercial motion tracking and virtual reality [8]. In the present study, we highlight the development of a motion-tracking system based on the use of inertial sensors and a distributed sensor network and show the functional feasibility of the rehabilitation system. This motion-tracking system consists of two inertial sensors attached to the upper limb of the subject and a motion-tracking algorithm running on a PC. The inertial sensor consists of a three-axis microelectromechanical system (MEMS) accelerometer, a gyroscope, and geomagnetic sensors. The sensor data are transmitted wirelessly to a PC, where the orientation and position of the subject are tracked. Virtual reality using the OpenGL library (http://www.opengl.org/) is implemented on the PC to provide rehabilitation movements and assess the upper extremity functionality of the subject. The system can be used at home without the assistance of a therapist. To verify the functional feasibility, this system was applied to healthy subjects and to subjects with a simulated stiff elbow condition. The stiffness was simulated by taping the elbow, which restricted its range of motion.

2. System Overview

The proposed system consists of a motion tracker and rehabilitation content (Figure 1). The motion tracker consists of two inertial measurement units (IMU) attached to segments of the subject's upper limb and a motion-tracking algorithm running on a PC. The IMUs are connected with a proprietary distributed sensor network. One IMU collects the sensor data of the other IMU and transmits both sensor data packets wirelessly to the PC. The motion-tracking algorithm on the PC receives the data packets, estimates the orientations of the IMSs from the sensor data, and finally calculates the segment positions of the subject along with the estimated orientation and segmental length. The rehabilitation content, which also runs on the PC, provides the trajectory of the upper limb using a graphic library and evaluates the subject's real trajectory.

Figure 1

System overview. The two IMUs and the motion-tracking algorithm utilize a motion tracker. The motion-tracking algorithm and rehabilitation content are implemented on a PC.

Each IMU (17.8 mm × 13.0 mm) is composed of a microcontrol unit (MCU), microelectromechanical system (MEMS) inertial sensors, and a Bluetooth communication module (Figure 2(a)). A three-axis accelerometer/magnetometer (LSM303DLHM, STMicroelectronics) and a three-axis gyroscope (L3GD20, STMicroelectronics) were used as the inertial sensors. The gyroscope was set to a full scale of ±500°/s and a sampling rate of 100 Hz. The MCU (STM32F103C8, STMicroelectronics) reads the sensors using interintegrated circuit (I2C) communication. The master and slave IMUs were attached to the upper arm and forearm of the subject, respectively. The communication between the master and slave IMUs was conducted using a distributed sensor network implemented based on another I2C communication. The I2C master and slave protocols were programmed in the master and slave IMUs, respectively. The MCU in the master IMU reads the sensor data of the slave IMU through I2C communication and sends the master and slave data packets using a Bluetooth module (Parani ESD200, Sena Technology, Korea) to the PC.

Figure 2

Inertial measurement unit (IMU): (a) sensor board and its direction and the (b) slave (left) and master (right) IMUs.

3. Motion-Tracking Algorithm

The transmitted data packet is processed using the motion-tracking algorithm on the PC to estimate the IMU's orientation. This orientation is expressed as a rotation matrix, $^{S 2 G} R$ , from the global frame to the sensor frame, which is calculated by integrating the gyroscope's angular velocity signal, $ω = {[ω_{x} ω_{y} ω_{z}]}^{T}$ [9]:

\begin{matrix} Δ R_{k}^{S2G} = I + \frac{\sin σ}{σ} [σ \times] + \frac{1 - \cos σ}{σ^{2}} {[σ \times]}^{2}, \end{matrix}

(1)

\begin{matrix} R_{k + 1}^{S2G} = R_{k}^{S2G} + Δ R_{k}^{S2G}, \end{matrix}

(2)

where

\begin{matrix} σ = ω \cdot Δ t {= [σ_{x} σ_{x} σ_{x}]}^{T}, \\ [σ \times] = [\begin{bmatrix} 0 & - ω_{z} & ω_{y} \\ ω_{z} & 0 & - ω_{x} \\ {- ω}_{y} & ω_{x} & 0 \end{bmatrix}] . \end{matrix}

(3)

However, small offsets in the gyroscope signal accumulate during the integration, which is known as a drift problem. Drift is corrected by fusing the gravity and Earth's magnetic field sensed using a Kalman filter [9, 10]. In particular, the system model for human-motion tracking is described well in [9]. For self-completeness, the orientation estimation can be summarized as follows (Figure 3).

Figure 3

Kalman filter structure. The gyroscope signal, $y_{G}$ , is integrated to predict the sensor orientation. The predicted orientation is corrected with the accelerometer signal, $y_{A}$ , and magnetometer signal, $y_{M}$ .

The accelerometer, gyroscope, and magnetometer sensor signals ( $y_{A, t}$ , $y_{G, t}$ and $y_{M, t}$ ) $y_{A, t}$ are modeled as in (1):

\begin{matrix} y_{A, t} = a_{t} - g + v_{A, t}, \\ y_{G, t} = ω_{t} + b_{t} + v_{G, t}, \\ y_{M, t} = m_{t} + d_{t} + v_{M, t}, \end{matrix}

(4)

where $a_{t}$ , $ω_{t}$ , and $m_{t}$ are the acceleration, gyroscope offset, and magnetic field, respectively; g, $b_{t}$ , and $d_{t}$ are gravity, the gyroscope offset, and magnetic disturbance; and $v_{A, t}$ , $v_{G, t}$ , and $v_{M, t}$ represent white Gaussian measurement noises.

The acceleration and magnetic disturbance are modeled into a Markov process model with white driving noise and a constant of between 0 and 1, as in

\begin{matrix} a_{t} = {c_{a} a}_{t - 1} + w_{a, t}, \\ d_{t} = {c_{d} d}_{t - 1} + w_{d, t} . \end{matrix}

(5)

In this indirect Kalman filter, the error state including the orientation, gyroscope offset, and magnetic distortion, $x_{ϵ, t} = {[θ_{ϵ, t} b_{ϵ, t} d_{ϵ, t}]}^{T}$ , is estimated. The error state transition matrix is defined from the error dynamics, as in

\begin{matrix} A = [\begin{bmatrix} I_{3 \times 3} & 0_{3 \times 3} & {- T \cdot I}_{3 \times 3} \\ 0_{3 \times 3} & I_{3 \times 3} & I_{3 \times 3} \\ 0_{3 \times 3} & 0_{3 \times 3} & I_{3 \times 3} \end{bmatrix}] . \end{matrix}

(6)

After the error state is estimated, it is added to the state estimation. With an estimated orientation of θ, included in the estimated state, the acceleration and magnetic signals in the global frame can be estimated. Since the estimated orientation has an error, these estimated signals also have an error. This estimated orientation is therefore corrected by comparing the estimated and measured signals with a measurement matrix, as in

\begin{matrix} C = [\begin{bmatrix} - Z_{G} \times & T Z_{G} \times & 0_{3 \times 3} \\ - H_{G} \times & T H_{G} \times & - c_{d} I_{3 \times 3} \end{bmatrix}], \end{matrix}

(7)

where $Z_{G}$ and $H_{G}$ correspond to the vertical Z acceleration and magnetic field in the global frame calculated by integrating the gyroscope signal, respectively.

The error state covariance Q and measurement covariance R matrixes are calculated as in

\begin{matrix} Q = [\begin{bmatrix} T^{2} Q_{v_{G}} & 0_{3 \times 3} & 0_{3 \times 3} \\ 0_{3 \times 3} & Q_{w_{b}} & 0_{3 \times 3} \\ 0_{3 \times 3} & 0_{3 \times 3} & Q_{w_{d}} \end{bmatrix}], \\ R = [\begin{bmatrix} T^{2} Q_{v_{G}} + Q_{v_{A}} & 0_{3 \times 3} \\ 0_{3 \times 3} & T^{2} Q_{v_{G}} + Q_{v_{M}} \end{bmatrix}], \end{matrix}

(8)

where $Q_{v_{G}}$ , $Q_{v_{G}}$ , and $Q_{v_{G}}$ are the measurement covariance matrixes of the gyroscope, accelerometer, and magnetometer. $Q_{v_{G}}$ and $Q_{v_{G}}$ correspond to the covariances of the gyroscope offset and magnetic disturbance.

With these matrixes, the indirect Kalman filter estimates the current state from the previous state and state transition matrix A. The covariance of the estimated state error P is also estimated as in (9). $P_{k + 1 | k}$ indicates the covariance of the state error estimated at time $k + 1$ , based on that at previous time k:

\begin{matrix} P_{k + 1 | k} = A_{k} P_{k | k} A_{k}^{T} + Q_{k} . \end{matrix}

(9)

Finally, the measurement is updated as in

\begin{matrix} v_{k + 1} = C_{k + 1} x_{k + 1} - z_{k + 1}, \\ K_{k + 1} = P_{k + 1 | k} C_{k + 1}^{T} {(C_{k + 1} P_{k + 1 | k} C_{k + 1}^{T} + R_{k + 1})}^{- 1}, \\ x_{ε_{k + 1 | k}} = K_{k + 1} v_{k + 1}, \\ P_{k + 1 | k + 1} = A_{k} P_{k + 1 | k + 1} A_{k + 1}^{T} + Q_{k + 1}, \end{matrix}

(10)

where $z_{k + 1}$ is a measurement consisting of acceleration and magnetometer signals, and $K_{k + 1}$ corresponds to the Kalman filter gain.

These procedures from (1) to (10) are repeated when the sensor signals are received. They are programmed with MATLAB scripts and converted into C files using a codegen tool (MATLAB7.12, Mathworks). The converted C files are included and compiled as a PC application, which includes the motion-tracking algorithm and rehabilitation content. The execution time of the orientation algorithm with a Kalman filter was measured to be about 0.03 ms on a PC (3.0 GHz dual-core Pentium with 2.0 GB of RAM).

With the estimated orientations and predefined segmental lengths, the positions of the upper arm $P_{upper}$ and forearm $P_{fore}$ relative to the shoulder joint were calculated using the forward kinematics. For the upper arm, the position was calculated as in

\begin{matrix} P_{upper} = R^{G2U} L_{upper}, \\ P_{fore} = {R^{G2F} R}^{G2U} L_{fore} + L_{upper}, \end{matrix}

(11)

where $R^{G2U}$ and $R^{G2F}$ are the rotations of the sensors in the upper arm and forearm, and $L_{upper}$ and $L_{fore}$ correspond to the measured segmental lengths of the upper arm and forearm, respectively.

4. Upper Extremity Rehabilitation Program

The upper extremity rehabilitation application was programmed using Visual C++ 6.0 (Figure 4). The virtual reality platform used to express three-dimensional movement of the upper extremity and the two-dimensional upper extremity rehabilitation program were implemented using OpenGL.

Figure 4

Virtual-reality-based upper extremity rehabilitation program. A conceptual drawing (a) and the simplified version used in this study (b).

The rehabilitation content was designed to provide both training and evaluation at the same time. The content included various dynamic movements and static holds and provided them to the subject within the virtual environment (Figure 5). The dynamic movements consisted of various movements such as shoulder flexion, abduction, adduction, and elevation; elbow flexion and extension; and forearm pronation and supination. The evaluation algorithm was derived from well-established methods including the Upper Extremity Motion Score [11] and Fugl-Meyer Assessment Scale [12]. The static holds were designed to maintain the arm position for a certain period after stretching and lifting the arms and were evaluated based on the National Institutes of Health (NIH) Stroke Scale [13] and the European Stroke Scale [14]. Based on these rationales, the functionality of each subject was evaluated for its spatiotemporal aspects. The average positional difference ( $D_{pos}$ ) between the provided training trajectory and the hand position of the patient was calculated as 100 Hz. The average time duration ( $T_{out}$ ) was also accumulated when the positional difference was larger than a predefined threshold.

Figure 5

Training content examples: (a) static hold, (b) linear movement, and (c) circular movement.

5. Feasibility Test

To verify its functional feasibility, the system was applied to five healthy subjects (26–30 years old, all male). To simulate a stiff elbow, which is a poststroke complication, the elbow of each healthy subject was taped to restrict its range of motion (Figure 6). The figure shows the system configuration for both the normal group and the group with a simulated stiff elbow [15, 16].

Figure 6

Subject conditions: (a) normal condition and (b) stiffness simulated by taping the elbow.

The linear and circular movements were provided within the virtual environment. The subjects were asked to follow the provided trajectories as best they could. The linear movements were provided in eight directions (Figure 5(b)), whereas the circular movements were provided in the CW and CCW directions (Figure 5(c)). Each movement was repeated ten times for each subject. The values of $D_{pos}$ and $T_{out}$ were calculated and analyzed.

Figure 7 shows the hand position trajectory for both the normal and stiff-elbow simulated groups. It was observed that the trajectory under a simulated stiff-elbow condition was qualitatively rougher than the trajectory under normal conditions.

Figure 7

Trajectory example: (a) linear and (b) circular movements. The black line, blue rectangles, and red circles correspond to the training trajectory provided, the tracked positions under normal conditions, and the tracked positions for the stiff-elbow simulated group, respectively.

Table 1 summarizes $D_{pos}$ and $T_{out}$ for the linear and circular movements. All $D_{pos}$ s and $T_{out}$ s values were merged over the subjects and movement types and compared using a paired t-test for the linear and circular movements of both groups, respectively. The values of $D_{pos}$ s and $T_{out}$ s showed significant differences between the normal and stiff-elbow simulated groups (all $P < 0.001$ for linear $D_{pos}$ , linear $T_{out}$ , circular $D_{pos}$ , and circular $T_{out}$ ). This demonstrated the functional feasibility of the proposed system for evaluating the upper extremity function of a poststroke patient in a quantitative manner.

Table 1

$D_{pos} s$ (cm) and $T_{out} s$ (s) for the two groups and movement type (cm): N, normal group; S, simulated stiff elbow group.

group	$T_{out}$		$D_{pos}$
group	Linear movement	Circular movement	Linear movement	Circular movement
N	0.07	0.97	5.40	5.76
S	0.39	3.79	5.89	6.76

6. Conclusion

In this study, an extremity rehabilitation program was proposed based on the use of IMUs and a virtual-reality system. The IMUs were used to track the subject's hand based on its calculated orientation and segment lengths with forward kinematics. In addition, the training content, including various dynamic movements and static holds, was designed to evaluate the spatiotemporal aspects of the subject's functionality. To quantitatively differentiate the evaluations, the proposed system was tested on both a group of healthy subjects and a group of subjects with a simulated elbow stiffness achieved by taping the elbow.

Using the upper extremity rehabilitation system developed in this study, we anticipate that patients will be able to undergo upper extremity rehabilitation on their own without the help of a therapist. Furthermore, a subsequent evaluation of the upper extremity function can be conducted by selecting an appropriate training program; this will increase the effectiveness of their rehabilitation. In future studies, the validity and reliability of upper extremity rehabilitation programs will be assessed for patients with a poststroke upper extremity hemiplegia. Three-dimensional upper extremity rehabilitation programs will be developed to ensure more accurate rehabilitation training for various upper extremity movements as displayed three dimensionally.

Footnotes

Acknowledgments

This work was supported by the Technology Innovation Program (project no. SA113286) funded by the Small & Medium Business Administration (SMBA, Korea) and by the Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Education, Science and Technology (2013-026506).

References

Willmann

R. D.

Lanfermann

Saini

Timmermans

Te Vrugt

Winter

Home stroke rehabilitation for the upper extremity

Proceedings of the 29th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBS '07)

August 2007

4015 4018

Parker

V. M.

Wade

D. T.

Langton Hewer

Loss of arm function after stroke: measurement, frequency, and recovery

International Rehabilitation Medicine 1986 8 2 69 73

2-s2.0-0022480548

Kuttuva

Boian

Merians

Burdea

Bouzit

Lewis

Fensterheim

The Rutgers arm, a rehabilitation system in virtual reality: a pilot study

Cyberpsychology and Behavior 2006 9 2 148 151

2-s2.0-33646743848

10.1089/cpb.2006.9.148

Mouawad

M. R.

Doust

C. G.

Max

M. D.

McNulty

P. A.

Wii-based movement therapy to promote improved upper extremity function post-stroke: a pilot study

Journal of Rehabilitation Medicine 2011 43 6 527 533

2-s2.0-79959497136

10.2340/16501977-0816

Lange

Chang

C.-Y.

Suma

Newman

Rizzo

A. S.

Bolas

Development and evaluation of low cost game-based balance rehabilitation tool using the microsoft kinect sensor

Proceedings of the 33rd Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBS '11)

August-September 2011

1831 1834

2-s2.0-84055184567

10.1109/IEMBS.2011.6090521

Pastor

Hayes

H. A.

Bamberq

S. J.

A feasibility study of an upper limb rehabilitation system using kinect and computer games

Proceedings of the 34th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBS '12)

August-September 2012

1286 1289

Zhang

Zhou

An interactive Internet-based system for tracking upper limb motion in home-based rehabilitation

Medical and Biological Engineering and Computing 2008 46 3 241 249

2-s2.0-39649112622

10.1007/s11517-007-0295-6

Kim

J. N.

Ryu

M. H.

Yang

Y. S.

Kim

T. K.

Upper extremity rehabilitation program using inertial sensors and virtual reality for patients with upper extremity hemiplegia due to disorders after stroke

Proceedings of International Conference on Computer Science and Technology (CST '12)

June 2012

71 76

MayBeck

P. S.

Stochastic Models, Estimation, and Control 1982 2

New York, NY, USA

Academic Press

10.

Roetenberg

Luinge

H. J.

Baten

C. T. M.

Veltink

P. H.

Compensation of magnetic disturbances improves inertial and magnetic sensing of human body segment orientation

IEEE Transactions on Neural Systems and Rehabilitation Engineering 2005 13 3 395 405

2-s2.0-26444563443

10.1109/TNSRE.2005.847353

11.

Bard

Hirschberg

G. G.

Recovery of voluntary motion in upper extremity following hemiplegia

Archives of Physical Medicine and Rehabilitation 1965 46 567 572

2-s2.0-0000349856

12.

Rödén-Jüllig

Britton

Gustafsson

Fugl-Meyer

Validation of four scales for the acute stage of stroke

Journal of Internal Medicine 1994 236 2 125 136

2-s2.0-0028101810

13.

Brott

Adams

H. P.

Jr. Olinger

C. P.

Marler

J. R.

Barsan

W. G.

Biller

Spilker

Holleran

Eberle

Hertzberg

Rorick

Moomaw

C. J.

Walker

Measurements of acute cerebral infarction: a clinical examination scale

Stroke 1989 20 7 864 870

2-s2.0-0024328055

14.

Hantson

de Weerdt

de Keyser

Diener

H. C.

Franke

Palm

van Orshoven

Schoonderwalt

De Klippel

Herroelen

Feys

The European stroke scale

Stroke 1994 25 11 2215 2219

2-s2.0-0028037069

15.

MacKean

L. C.

Bell

Burnham

R. S.

Prophylactic ankle bracing versus taping: effects on functional performance in female basketball players

Journal of Orthopaedic & Sports Physical Therapy 1995 22 2 77 81

16.

Patterson

Caulfield

Using a shoe mounted tri-axial accelerometer to detect kinematic changes during stiff ankle walking

Proceedings of the Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBS '11)

August 2011

3492 3495