Upper limb kinematic measurement using markerless motion capturing (MMC) in stroke survivors: A cross-sectional experimental study

Abstract

Introduction

Markerless motion capture (MMC) technology has emerged as a clinical tool to assess the physical performance of patients. This study evaluates: (a) differences in upper limb joint angles between stroke survivors with different functional levels and their healthy counterparts in controlled indoor and uncontrolled outdoor environments; and (b) the relationship between the kinematic information obtained by the MMC system and the scores of manual motor assessments.

Methods

A customized MMC system using an iPad Pro captured the participants’ movements. Stroke survivors underwent three upper limb assessments and performed seven sets of upper limb tasks with their non-affected side, followed by their affected side. Healthy participants performed the same tasks with their dominant and non-dominant sides. The sensitivity and specificity of the machine models were calculated for classifying upper limb motor function levels using kinematic data from the MMC system.

Results

Fifty stroke survivors and 49 healthy adults were recruited. Significant differences were found between the affected and non-affected sides of stroke participants in most tasks. Significant positive correlations were found between the results of the manual motor assessments and most of the kinematic parameters. The results of the four selected machine learning models revealed ≥0.85 sensitivity in the stroke upper limb functional level classification.

Conclusion

The MMC system and machine learning algorithms provide accurate data for evaluating upper limb recovery in stroke survivors. Further research is needed to explore the use of the MMC system by stroke survivors at home during remote therapy.

Keywords

Kinematic assessment upper limb markerless motion capture stroke

Introduction

The motor recovery of stroke survivors often requires continuous monitoring by therapists to customize exercises for different recovery stages.¹ Conventional practices undertaken by therapists, such as regular functional assessments and the manual measurement of range of motion (ROM), require patients’ regular attendance in clinical settings.² The use of marker-based motion analysis systems and other kinematic evaluation techniques—such as extracting data like joint angles and movement counts from wearable sensors to quantify stroke rehabilitation progress—is often costly and time-consuming to set up and implement.³ With advances in technology, markerless motion capture (MMC) technology has emerged as a clinical measurement tool that can be used to assess the physical performance of patients more naturally, as well as reducing time-consuming tasks in manual measurements conducted by therapists.⁴ It also has the advantage over wearable sensors or complex optoelectronic laboratory setups for motion analysis while maintaining high kinematic accuracy.^5–7 A previous study has been conducted to capture kinematic data regarding stroke survivors and healthy adults when performing tasks through a stroke-specific performance-based impairment index: the Fugl–Meyer Assessment (FMA). Movement data from stroke survivors and healthy adults can be successfully classified with a rate of above 90% using machine learning classification models.⁸ Therefore, the MMC system has the potential to identify symptomatic movement patterns in stroke survivors through artificial-intelligence-assisted detection technology, in order to monitor patients’ motor performance and activities of daily living, especially during remote assessments as part of telerehabilitation.⁹

While MMC systems facilitate the tracking of movement kinematics, it is difficult for therapists to interpret this kinematic data into a comprehensive understanding of patients’ actual motor functions.¹⁰ Researchers have also proposed using mobile devices as MMC systems for remote rehabilitation assessments, which enhances accessibility by eliminating the need for additional hardware.¹¹ However, little research has been carried out to investigate the application of MMC systems in mobile devices for clinical motor assessments among stroke survivors.^12,13 Since most of the MMC systems have been applied only to healthy subjects, the validity and efficacy of the MMC system in classifying the motor impairment of various severities in the clinical populations remains unclear. Moreover, there are limited studies to standardize the use of MMC for clinical interpretation and translate the kinematic data captured by the MMC to be used together with clinical assessment scales such as the FMA. These hence create a large research gap concerning the application of mobile MMC systems, particularly regarding their efficacy in providing accurate measurements with which to evaluate the motor performance of stroke survivors with varying levels of severity. Our pilot study demonstrated the good reliability of a customized MMC system on an iPad Pro for measuring active range of motion (AROM) and angular waveforms in healthy adults.¹⁴ This system might help to measure the motor performance of stroke survivors who demonstrate movement impairment; this study therefore further evaluates the performance of the customized MMC system in detecting : (a) the differences in upper limb joint angles between stroke survivors with different motor functional levels and their healthy counterparts in both controlled indoor and uncontrolled outdoor environments; and (b) the relationship between the kinematic information obtained by the MMC system and the scores of manual motor assessments. This study also investigates the sensitivity and specificity of the classification of upper limb motor functional level using machine learning methods, based on the kinematic data from the MMC system on the mobile device.

Methods

Study design

This cross-sectional experimental study received ethical approval from the Human Subjects Research Ethics Committee of the Hong Kong Polytechnic University (HSEARS20230214010). All subjects provided informed written consent before participating.

Stroke survivors completed three upper limb assessments conducted by an occupational therapist: (a) the Functional Test for the Hemiplegic Upper Extremity (FTHUE), which evaluates task-specific arm function¹⁵; (b) the FMA for the Upper Extremity (FMA-UE), which is a comprehensive measure of motor recovery¹⁶; and (c) the Wolf Motor Function Test (WMFT), which quantifies movement quality and speed during functional tasks.¹⁷ Participants performed seven upper limb tasks from stroke-specific assessments, starting with their non-affected side, followed by their affected side. Assuming that healthy participants would score full marks in all assessments, they instead performed the same sets of tasks for motion capturing with their dominant side, followed by their non-dominant side. All participants completed the tasks in a controlled indoor environment and in three randomly selected uncontrolled outdoor environments immediately after. Data from the MMC system were analyzed using t-tests, ANOVAs, and correlation metrics to compare kinematic between the participants with stroke and their healthy counterparts. Machine learning models were implemented to predict FTHUE and FMA-UE classifications by using the kinematic features.

Instrumentation

A customized MMC system developed using Xcode on the basis of the ARKit6 and RealityKit framework in an iPad Pro with a LiDAR scanner, which had undergone reliability testing for upper limb kinematic measurement,¹⁴ was used to capture participants’ movements in this study. The detection of the human body and the joint positions were extracted and visualized through computer-vision algorithms using convolutional neural networks (CNNs). A total of 14 3D body-joint positions and the timestamp of the motion detection were captured by our motion-tracking platform. The capture frequency of the MMC system was set at 30 Hz. The normalized coordinates of the angles were relative to the center of the pelvis and defined as the origin of the ARKit's coordinate system. The adjacent 3D joint coordinates extractions were used to calculate the angles of interest (AOI). Angle $θ$ was calculated by the three joints—shoulder, elbow, and wrist—namely, $A, B, C \in R^{3}$ or associated vectors ${\vec{v}}_{1} = A - B and {\vec{v}}_{2} = C - B,$ with the formula:

θ = \arccos \frac{v_{1} \cdot v_{2}}{{| | v_{1} | |}_{2} {| | v_{2} | |}_{2}}

Sample size calculation

We assumed a two-tailed comparison with a type I error rate at 0.05, with 80% power. The stroke survivors were stratified according to the lower and higher functioning of their upper limb performance using the FTHUE.¹⁵ As a conservative estimation with a discard rate of 15% due to bad data or outliers, according to our previous pilot study,¹⁴ we presumed that 42 subjects in both healthy and stroke groups would be required for the final data analysis. Power analysis conducted with GPower 3.1.9.2 calculated the effect size as 0.74, which is between medium (0.5) and large (0.8).¹⁸

Participants

Stroke survivors were recruited from community self-help groups and healthy counterparts through convenience sampling.

Inclusion and exclusion criteria for the participants

Inclusion criteria

To be eligible to take part in the study, participants were included if: (a) they were adults aged 18 years or above; (b) they had been diagnosed with a hemiplegic stroke; (c) they did not have a history of previous neurological or orthopedic diseases or congenital disorders of the upper or lower extremities or the spine; (d) they possessed adequate cognitive ability to understand instructions; and (e) they were able to engage in a 1-hour experimental session.

Exclusion criteria

Participants who met the following conditions were excluded: (a) medically unstable; (b) previous injuries or medical conditions over the upper limbs or spine affecting their upper limb functions (for healthy participants); (c) stroke survivors with a functional level of two or below, as measured using the FTHUE.

Motion capture tasks

In the motion capturing experiment, participants performed seven sets of upper limb tasks. The tasks included: (a) Task 3 in the FMA-UE, bringing the hand to the same side of the ear; (b) Task 4 in the FMA-UE, extending the arm to the opposite knee; (c) Task 6 in the FMA-UE, with shoulder flexion to 90° with the elbow at 0°; (d) Task 8 in the FMA-UE, with shoulder abduction 0° to 90° with the elbow fully extended and the forearm pronated; (e) Task 9 in the FMA-UE, with shoulder flexion beyond 90° with the elbow at 0° and the forearm in the mid position; (f) Task 3D in the FTHUE, holding a pouch; and (g) Task 24 in the FMA-UE, which is a finger-to-nose test. All of the tasks in this session were repeated five times. The stroke survivors were instructed to perform each task with their unaffected side first, followed by their affected side. Figure 1(a) to (g) illustrates the desired postures for these tasks.

Figures 1.

(a–g) Left to right, top to bottom, the desired postures for Task 1 to Task 7.

Experiment setup

Controlled indoor environment

The experiment was conducted at the university laboratory, where the floor was covered with vinyl to prevent it from being slippery. For the motion capturing session, participants stood in front of a plain wall in the same laboratory. One iPad Pro was placed two meters in front of the participant, and another two iPad Pros were placed at the lateral left and right sides of the participants, respectively. All iPads were connected to the same local network, and the motion capture system was launched on each device to establish a handshake signaling. At the start of recording, the master iPad sent a timestamped “start” command, triggering all devices to begin logging data with synchronized timestamps. Post-processing alignment of the data streams was performed using the shared timestamps to ensure temporal precision.

Uncontrolled outdoor environment

Three open areas at the university were chosen as uncontrolled outdoor environments. A 2.5 m × 2.5 m area was marked. Two 1 m × 1 m anti-slip mats were placed on both sides of the participants to prevent them from slipping. Three iPad Pros placed on a tripod stand were brought to the locations. The iPad Pro placement was the same as that in the laboratory environment.

Statistical analysis

Kinematic data, including completion time and joint angles in the target positions (ATP), were extracted from the MMC system, excluding the first trial. The normality of data distribution was assessed using the Shapiro–Wilk test to ensure the data meet the assumptions for parametric analysis. Averages from the second to fifth trials were analyzed using independent t-tests and ANOVAs to compare joint angles and completion times between stroke populations and healthy subjects as well as among the stroke survivors with different functional levels. Differences in angular waveforms between the two sides were assessed with the coefficient of multiple correlation (CMC) and root mean square error (RMSE). Pearson's r correlation was used to evaluate relationships among assessment results and kinematic data (Appendix 1).

Four machine learning models, including logistic regression (LG), a Naive Bayes (NB) classifier, a support vector machine (SVM), and a decision tree (DT) model were used to investigate the trajectory in the predictions of the stroke participants’ upper limb assessment results, specifically the impairment levels according to the FMA-UE scores and the functional levels according to the FTHUE scores, based on kinematic data from the MMC system—the ATP and differences between the affected and unaffected sides. Regarding the FTHUE classification, participants with stroke were stratified into two groups: lower functioning (levels 3–4) and higher functioning (levels 5–7) group.¹⁹ Regarding the FMA-UE classification, patients were categorized into three groups based on FMA-UE scores: higher functioning (FMA-UE > 52), moderate functioning (31 ≤ FMA-UE ≤ 52), and lower functioning (FMA-UE < 31).²⁰ The dataset was divided into training and test splits, using five-fold subject-wise stratified cross-validation.²¹ These machine learning models were evaluated using standard classification and regression performance metrics. Regarding the FTHUE and FMA-UE groups prediction, the sensitivity, specificity, and AUC of each classification models were assessed.

All of the statistical tests were performed using IBM SPSS 26, while the CMC and RMSE values were generated by MATLAB R2020a. All machine learning models were run using the Scikit-learn package in Python.

Results

Fifty stroke survivors and 49 healthy adults were recruited. The mean age of the stroke survivors and the healthy adults was 58.9 years (SD: 11.7) and 60.2 years (SD: 8.5), respectively. Detailed demographic information regarding the participants is presented in Table 1.

Table 1.

Demographic information of the participants.

	Stroke group	Healthy group
Mean age	58.9 (11.7)	60.2 (8.5)
Gender ratio (male:female)	32:18	18:31
FTHUE Functional level (n)
Levels 3–4	18	NA
Levels 5–7	32	NA
Affected side (n)
Right	22	NA
Left	28	NA
Dominant hand (Pre-onset) (n)
Right	49	48
Left	1	1

Note: FTHUE: Functional Test for the Hemiplegic Upper Extremity.

The shoulder and elbow joint angles at the targeted positions (ATP) for Tasks 1 to 6, are shown in Figure 2(a) and (b). Significant differences were found between the affected and non-affected sides of stroke survivors with lower functioning in all tasks, except for the shoulder angles in Tasks 1 and 2 (MD = −3.0 and −4.1). For stroke survivors with higher upper limb functioning, significant differences were observed in all tasks except for the shoulder angles in Tasks 3 and 5 (MD = −2.2 and −5.0). No significant differences were found between the two sides of healthy participants, except for shoulder and elbow angles in Task 3 (MD = 11.5 and 4.2) and elbow angles in Task 6 (MD = 14.8). Comparisons of the affected sides of higher and lower functioning stroke survivors showed significant differences in all tasks except for shoulder angles in Task 2 (p = 0.11). Significant differences were found between the affected side of lower functioning stroke survivors and their healthy counterparts in all tasks except for shoulder angles in Tasks 1 and 2 (p = 0.827 and p = 0.264, respectively). Significant differences were also found between higher functioning stroke participants and healthy participants in all tasks, except for shoulder angles in Tasks 2 and 5 (p = 0.282 and 0.229, respectively).

Figure 2.

(a) Comparison of the ATP of the shoulder in Task 1 to Task 6. (b) Comparison of the ATP of the elbow in Task 1 to Task 6.

Differences in angular waveforms between lower and higher functioning stroke groups and healthy participants were calculated using CMC and RMSE, as shown in Table 2. Lower functioning stroke survivors had CMC values ranging from 0.23 to 0.67 across all tasks. In contrast, higher functioning stroke survivors had CMC values ranging from 0.39 (elbow in Task 1) to 0.86 (elbow in Task 4), while healthy participants ranged from 0.80 to 0.92 in Tasks 1 to 4.

Table 2.

CMC and RMSE values of the angular waveform comparison between the two sides.

Task	Stroke (lower functioning group)		Stroke (higher functioning group)		Healthy group
Task	CMC (SD)	RMSE (SD)	CMC (SD)	RMSE (SD)	CMC (SD)	RMSE (SD)
Task 1Bring hand to ear
Shoulder	0.53 (0.08)	22.3 (9.3)	0.68 (0.06)	26.9 (18.6)	0.89 (0.10)	8.2 (5.2)
Elbow	0.31 (0.12)	38.7 (13.8)	0.39 (0.07)	15.2 (6.5)	0.88 (0.07)	10.5 (3.3)
Task 2Hand to opposite knee
Shoulder	0.67 (0.09)	12.1 (7.7)	0.65 (0.11)	16.1 (9.3)	0.92 (0.05)	5.8 (2.4)
Elbow	0.38 (0.08)	37.4 (10.0)	0.53 (0.06)	13.8 (5.6)	0.89 (0.08)	9.4 (5.2)
Task 3Shoulder flexion to 90°
Shoulder	0.51 (0.06)	28.9 (15.2)	0.78 (0.05)	26.3 (12.3)	0.78 (0.07)	15.7 (8.2)
Elbow	0.29 (0.05)	33.6 (18.4)	0.61 (0.10)	24.2 (15.9)	0.88 (0.11)	11.3 (6.5)
Task 4Shoulder flexion to 180°
Shoulder	0.25 (0.11)	41.2 (19.0)	0.62 (0.09)	21.2 (12.7)	0.87 (0.05)	14.4 (7.2)
Elbow	0.27 (0.07)	28.9 (11.4)	0.86 (0.13)	19.9 (9.3)	0.92 (0.03)	11.2 (6.8)
Task 5Shoulder abduction to 90°
Shoulder	0.56 (0.10)	22.7 (13.9)	0.71 (0.11)	14.1 (6.8)	0.88 (0.11)	15.2 (8.6)
Elbow	0.27 (0.08)	35.8 (16.2)	0.53 (0.15)	23.2 (10.0)	0.91 (0.05)	9.1 (5.4)
Task 6Hold a pouch (for 10 s)
Shoulder	0.30 (0.09)	33.6 (14.5)	0.67 (0.10)	18.4 (7.9)	0.82 (0.06)	10.4 (7.7)
Elbow	0.46 (0.11)	28.9 (9.2)	0.80 (0.11)	11.2 (5.2)	0.83 (0.09)	8.6 (4.8)
Task 7Finger-to-nose test
Shoulder	0.39 (0.10)	27.8 (12.7)	0.49 (0.13)	20.9 (11.4)	0.83 (0.05)	10.1 (5.3)
Elbow:	0.23 (0.06)	43.2 (22.6)	0.44 (0.07)	16.5 (9.7)	0.80 (0.09)	7.8 (4.2)

Note: CMC: coefficient of multiple correlation; RMSE: root mean square error.

Correlations between the kinematic data and assessment results (FTHUE, FMA-UE, WMFT) are summarized in Table 3. Significant correlations were found for most kinematic parameters. The elbow ATP of the affected side in Task 6 (“hold a pouch”) had the strongest positive correlations with FTHUE, FMA-UE, and WMFT (r = 0.944, 0.883, and 0.873, respectively). In contrast, kinematic data such as the ATP of the affected side, the ATP difference between sides, and the CMC and RMSE values from Task 2 (hand to opposite knee) showed weaker correlations with the assessment scores.

Table 3.

Correlation between the kinematic data and the assessment results of the FTHUE, FMA-UE, and WMFT.

Task	Stroke (All)
	ATP affected side r	ATP difference (non-affected – affected)	CMC (non-affected – affected)	RMSE (non-affected – affected)	Completion time	Bilateral difference in completion time
Correlation with FTHUE
Task 1Bring hand to ear
Shoulder	−0.046	−0.085	0.691*	0.322*	NA	NA
Elbow	−0.631*	0.496*	0.374*	−0.708*	NA	NA
Task 2Hand to opposite knee
Shoulder	0.273	−0.399*	−0.27	0.258	NA	NA
Elbow	0.398*	−0.313*	0.588*	−0.742*	NA	NA
Task 3Shoulder flexion to 90°
Shoulder	0.721*	−0.636*	0.809*	−0.159	NA	NA
Elbow	0.778*	−0.732*	0.793*	−0.091	NA	NA
Task 4Shoulder flexion to 180°
Shoulder	0.698*	−0.582*	0.818*	−0.590	NA	NA
Elbow	0.808*	−0.796*	0.778*	−0.358	NA	NA
Task 5Shoulder abduction to 90°
Shoulder	0.684*	−0.666*	0.393*	−0.264	NA	NA
Elbow	0.639*	−0.653*	0.578*	−0.201	NA	NA
Task 6Hold a pouch (for 10 s)
Shoulder	0.683*	−0.556*	0.727*	−0.677*	NA	NA
Elbow	0.944*	−0.839*	0.658*	−0.649*	NA	NA
Task 7Finger-to-nose test
Shoulder	NA	NA	0.322*	−0.638*	−0.655*	0.647*
Elbow	NA	NA	0.641*	−0.581*	−0.655*	0.647*
Correlation with the FMA-UE
Task 1Bring hand to ear
Shoulder	−0.054	−0.015	0.622*	0.247	NA	NA
Elbow	−0.603*	0.445*	0.231	−0.606*	NA	NA
Task 2Hand to opposite knee
Shoulder	0.301*	−0.415*	−0.039	0.261	NA	NA
Elbow	0.364*	−0.306*	0.497*	−0.694*	NA	NA
Task 3Shoulder flexion to 90°
Shoulder	0.668*	−0.593*	0.749*	0.307*	NA	NA
Elbow	0.716*	−0.660*	0.755*	0.310*	NA	NA
Task 4Shoulder flexion to 180°
Shoulder	0.694*	−0.607*	0.748*	0.092	NA	NA
Elbow	0.760*	−0.755*	0.675*	0.339*	NA	NA
Task 5Shoulder abduction to 90°
Shoulder	0.672*	−0.663*	0.310*	0.010	NA	NA
Elbow	0.584*	−0.589*	0.515*	0.365*	NA	NA
Task 6Hold a pouch (for 10 s)
Shoulder	0.618*	−0.489*	0.695*	0.168	NA	NA
Elbow	0.883*	−0.760*	0.599*	0.124	NA	NA
Task 7Finger-to-nose test
Shoulder	NA	NA	0.269	−0.432*	−0.602*	0.594*
Elbow	NA	NA	0.599*	−0.365*	−0.602*	0.594*
Correlation with the WMFT
Task 1Bring hand to ear
Shoulder	−0.12	−0.027	0.619*	0.262	NA	NA
Elbow	−0.564*	0.461*	0.237	−0.601*	NA	NA
Task 2Hand to opposite knee
Shoulder	0.214	−0.313*	−0.48	0.266	NA	NA
Elbow	0.291*	−0.216	0.504*	−0.730*	NA	NA
Task 3Shoulder flexion to 90°
Shoulder	0.643*	−0.572*	0.741*	−0.091	NA	NA
Elbow	0.716*	−0.683*	0.732*	−0.115	NA	NA
Task 4Shoulder flexion to 180°
Shoulder	0.688*	−0.606*	0.751*	−0.521*	NA	NA
Elbow	0.746*	−0.737*	0.644*	−0.303*	NA	NA
Task 5Shoulder abduction to 90°
Shoulder	0.664*	−0.653*	0.257	−0.184	NA	NA
Elbow	0.562*	−0.562*	0.565*	−0.108	NA	NA
Task 6Hold a pouch (for 10 s)
Shoulder	0.605*	−0.494*	0.671*	−0.655*	NA	NA
Elbow	0.873*	−0.728*	0.611*	−0.624*	NA	NA
Task 7Finger-to-nose test
Shoulder	NA	NA	0.239	−0.214	−0.611*	0.594*
Elbow	NA	NA	0.605*	−0.618*	−0.611*	0.594*

Note: ATP: angles in the target positions; CMC: coefficient of multiple correlation; FMA-UE: Fugl–Meyer Assessment for the Upper Extremity; FTHUE: Functional Test for the Hemiplegic Upper Extremity; RMSE: root mean square error; WMFT: Wolf Motor Function Test.

p < 0.05.

Four machine learning models—LG, SVM, NB classifier, and DT—were trained to classify upper limb function according to FTHUE levels, based on kinematic data from the MMC system (Table 4). Models using the ATP of the affected side in Tasks 1 to 6 and completion time in Task 7 achieved sensitivities of ≥0.85, with the LG model showing the highest sensitivity and specificity (0.94). Models using ATP differences between affected and non-affected sides achieved sensitivities of 0.89 (DT model) to 0.97 (SVM model), with all models having an AUC ≥ 0.86. Feature importance analysis indicated that bilateral ATP differences in the shoulder and elbow during Tasks 3, 4, and 5 were key predictors of upper limb function. Results were consistent across cross-validation, with an average accuracy of 86.3% (SD = 2.7%). The SVM model classified participants into four functioning levels according to FMA-UE scores. Table 5 summarizes the SVM coefficients and optimal cutoff scores, with Tasks 3, 4, and 5 ranked highest. The cutoff scores showed over 90% sensitivity for the three stroke functioning groups based on the bilateral elbow angle difference in Task 5.

Table 4.

Classification performance of the machine learning models (classification of lower and higher functioning level according to the FTHUE).

Model	Sensitivity (95% CI)	Specificity (95% CI)	AUC (95% CI)
By ATP
LG	94.6% (75.7%–98.4%)	94.3% (73.2%–96.2%)	0.94 (0.83–0.98)
NB	91.2% (83.2%–95.1%)	92.8% (69.3%–93.2%)	0.91 (0.80–0.94)
SVM	93.4% (71.3%–98.1%)	91.0% (69.2%–93.8%)	0.91 (0.81–0.92)
DT	85.2% (65.6%–88.4%)	87.3% (66.3%–89.4%)	0.86 (0.73–0.90)
By ATP difference between the two sides
LG	96.3% (80.1%–98.6%)	96.5% (78.4%–98.4%)	0.97 (0.83–1.00)
NB	94.1% (79.8%–96.7%)	93.0% (71.4%–94.3%)	0.93 (0.79–0.96)
SVM	97.1% (84.2%–98.6%)	96.5% (77.3%–97.9%)	0.97 (0.88–1.00)
DT	89.2% (80.0%–91.2%)	90.8% (67.0%–93.4%)	0.90 (0.77–0.93)

Note: ATP: angles in the target positions; NB: Naive Bayes; DT: decision tree; LG: logistic regression; SVM: support vector machine.

Table 5.

Optimal cutoff score, coefficient, sensitivity, and specificity for each parameter in SVM classification (according to the FMA-UE).

Parameter (joint/completion time)	Task	Optimal cutoff score(degree of angle differences between two sides)	Sensitivity (recall) (%)	Specificity (%)	SVM weight(coefficient)
Shoulder	Task 1				0.34
	Lower functioning group	5.4	79.8	75.3
	Moderate functioning group	12.6	81.4	80.3
	Higher functioning group	15.3	88.7	82.4
	Task 2				0.12
	Lower functioning group	2.9	54.1	47.3
	Moderate functioning group	5.6	55.8	52.0
	Higher functioning group	7.3	63.7	72.1
	Task 3				−4.26
	Lower functioning group	42.1	92.3	89.6
	Moderate functioning group	15.0	86.6	83.7
	Higher functioning group	7.7	88.9	85.8
	Task 4				−3.13
	Lower functioning group	39.9	92.6	88.5
	Moderate functioning group	28.3	92.5	83.1
	Higher functioning group	11.7	89.7	87.4
	Task 5				−5.17
	Lower functioning group	30.8	89.2	84.5
	Moderate functioning group	13.5	90.0	87.2
	Higher functioning group	8.5	88.8	90.1
	Task 6				−3.82
	Lower functioning group	40.1	88.2	85.7
	Moderate functioning group	21.1	86.7	89.1
	Higher functioning group	10.5	85.2	83.5
Elbow	Task 1				−0.66
	Lower functioning group	30.8	77.5	71.8
	Moderate functioning group	12.1	79.9	75.7
	Higher functioning group	5.4	73.3	70.1
	Task 2				−0.75
	Lower functioning group	19.1	88.3	80.2
	Moderate functioning group	7.1	71.8	79.4
	Higher functioning group	5.6	77.5	74.2
	Task 3				−4.35
	Lower functioning group	40.2	93.9	91.0
	Moderate functioning group	20.3	87.2	88.4
	Higher functioning group	10.8	89.1	82.3
	Task 4				−5.03
	Lower functioning group	43.4	93.4	82.1
	Moderate functioning group	22.4	83.2	90.5
	Higher functioning group	12.2	95.6	91.2
	Task 5				−6.14
	Lower functioning group	59.1	97.6	93.6
	Moderate functioning group	24.3	90.5	96.3
	Higher functioning group	11.9	92.6	89.2
	Task 6				−5.77
	Lower functioning group	53.7	97.5	96.3
	Moderate functioning group	20.8	91.6	86.8
	Higher functioning group	13.1	89.5	95.2
Completion time (s)	Task 7				−1.52
	Lower functioning group	5.36	95.6	92.5
	Moderate functioning group	2.83	88.2	81.3
	Higher functioning group	2.10	83.1	87.3

Note: SVM: support vector machine; FMA-UE: Fugl–Meyer Assessment for the Upper Extremity; lower functioning group (FMA-UE < 31); moderate functioning group (31 ≤ FMA-UE ≤ 52); higher functioning group (FMA-UE > 52).

The kinematic data captured by the MMC system in the outdoor environment contained a significant number of noise signals and missing data points, which hindered the formation of a complete angular waveform. Over half of the data had to be discarded due to noise signals. Due to the significant amount of outdoor data discarded, analysis of the outdoor data could not be performed. Figure 3(a) and (b) depicts the angular waveform extracted from a participant performing two tasks outdoors, demonstrating the noise signals and missing data points captured by the MMC system in the outdoor area. Figure 3(c) depicts the complete angular waveform of the same participants performing the task indoors.

Figure 3.

(a) Angular waveform of one participant performing Task 3 in an outdoor area. (b) Angular waveform of one participant performing Task 4 in an outdoor area. (c) Angular waveform of one participant performing Task 4 in an indoor area.

Discussion

In this study, we found that the MMC system detected significant differences in joint angles at task completion between the affected and non-affected sides of stroke survivors in all selected tasks, except for shoulder angles in the “bring hand to the same side of the ear” task among those with lower upper limb function. The affected side showed significant limitations in shoulder and elbow ranges, likely due to limited control, spasticity, or muscle weakness post-stroke.²² There were no significant differences between the dominant and non-dominant hands of healthy participants in most tasks. The CMC and RMSE values indicated greater differences between the affected and non-affected sides in stroke survivors with lower upper limb functioning compared to those with higher functioning. Our findings suggest that the MMC system in mobile devices is sensitive to capturing the kinematic differences between affected and non-affected sides, indicating its potential to provide kinematic information that distinguishes upper limb function levels in stroke survivors. The MMC system on a mobile device might hence serve as an alternative screening tool for motor ability in post-stroke patients, given that it can effectively detect symptomatic movements based on ROM differences and movement pattern variations.

All the selected classifiers achieved sensitivities above 0.84 based on joint angles from the affected side in lower and higher functioning classification, based on the FTHUE (FTHUE level five or above = higher functional level; FTHUE level four or below = lower functional level).²³ Our results indicate that combining the MMC system with machine learning can effectively distinguish between higher and lower levels of upper limb impairment in stroke patients.²⁴ Additionally, our AI models, trained on performance differences between affected and non-affected limbs, showed sensitivities above 0.89 and specificities of at least 0.90, indicating excellent performance.²⁵ Classifying based on these performance differences yielded higher sensitivity than considering only the affected side. One possible explanation for this result is that the non-affected hand typically reflects how an individual normally performs a motor task. Thus, comparing angular differences between the affected and non-affected sides may indicate how much the hemiparetic limb deviates from normal performance. A smaller difference suggests better recovery of motor function on the affected side and hence a higher functional level in classification.

In addition to FTHUE classification, FMA-UE scores were used to evaluate intervention effectiveness which supports the evidence-based practices for therapists. The SVM classifier identified optimal cutoff points for predicting FMA-UE performance, revealing that the lower functioning group had bilateral elbow angle differences exceeding 20°, while the higher functioning group had differences under 15° in most of the selected tasks. The SVM model highlighted that Tasks 3, 4, and 5, which involve specific elbow positions during shoulder movements, significantly contribute to functional classification. In contrast, tasks with minimal upper limb activity, such as the “hand to opposite knee” task, were less predictive of FMA-UE scores. Therefore, we recommend careful selection or combination of tasks for analyzing stroke survivors’ hemiplegic recovery, especially when comparing the non-affected side. The machine learning model suggests that combining the MMC system with classifiers can effectively distinguish upper limb function in stroke patients through a limited number of tasks, which can hence facilitate rapid screening of motor function in monitoring rehabilitation progress.

Although only seven tasks from the standardized upper limb assessment were selected, they are representative of the common functional tasks in standardized upper limb assessments, such as the FTHUE, FMA-UE, and WMFT. These seven tasks were chosen in this study due to their nature of being straightforward to administer, making them suitable for home settings where resources and supervision are limited. In addition, our tracking algorithm only included large joints in order to test its ability to identify and analyze participants’ gross motor abilities; mainly gross movements performed by the shoulder and elbow joint angles were investigated in this study. Future studies could consider tracking more complicated features, such as the contour of the hand and fingertips, so as to determine the ability of the MMC system to capture and analyze the movement of the wrist and the fine motor ability of stroke survivors.

Our findings indicate that kinematic data captured by the MMC system outdoors is affected by noise, background clutter, and light intensity.²⁶ The MMC often loses track of the target participant's joints when pedestrians pass by, misidentifying the pedestrian's limbs as the limbs of the target participant. A cluttered background can also confuse the system, which sometimes interprets tree branches as human limbs. Dim lighting results in sparse waveforms with missing data points as insufficient light hampers the capture of clear images, which affects tracking accuracy. We therefore recommend positioning the MMC in front of a plain background with adequate lighting and no moving objects to enhance data quality. Modifying the feature extraction function and including an initial calibration step may further improve joint identification and tracking accuracy.

The overall results of our study are consistent with previous recommendations by Bonnechère and colleagues²⁷ that an MMC system could be utilized to evaluate the upper limb motor performance of stroke survivors. Although it might not be appropriate for motion capturing in outdoor areas with a cluttered background, the kinematic data captured in a structured indoor environment provides a high level of sensitivity in regard to upper limb function classification. Although our study employed three iPad Pro devices to capture kinematic information from multiple angles, stroke survivors have the flexibility to utilize a single iPad Pro or their personal mobile device for motion capturing in a home setting. The use of three iPad Pro devices was solely intended to capture movements from diverse perspectives, while individuals can easily adjust the capturing angle independently when employing a single mobile device. Together with its portability, user-friendly setup, and low cost, an MMC system on a mobile device can facilitate remote monitoring of motor recovery during telerehabilitation.²⁸ Additionally, the precise data collected could enable therapists to conduct regular screenings of patients’ functional abilities at home, reducing the need for frequent clinic visits.

Even though current studies support the utilization of MMC technology in telerehabilitation, researchers must close the gap between research findings and the real-life implementation of MMC technology in order to promote its actual adoption in remote rehabilitation programs in the future. Future research should focus on designing a user-friendly interface that enables patients to operate the MMC system and transmit data to therapists effectively.

Limitations

This study employs data from 49 healthy adults and 50 stroke survivors. The sample size is considered small for training and testing machine learning models. Second, the ratio of stroke survivors with different functioning levels was not balanced. Future experiments examining the actual effect of different light levels on motion tracking quality in MMC systems are still warranted.

Conclusion

This study utilizes an MMC system on a mobile device to detect significant differences in the affected upper limbs of stroke survivors and healthy adults. The data provided by the MMC system reflects significant kinematic differences between the stroke survivors with lower upper limb functioning and those with higher functioning in all of the selected tasks. Strong correlations were found between upper limb motor assessment scores and kinematic performance. Combining the MMC system with a machine learning classification algorithm can provide precise data for evaluating upper limb recovery during telerehabilitation. Further research on the use of MMC systems by patients in home settings is warranted.

Footnotes

Abbreviations

Author note

Study was performed in the Hong Kong Polytechnic University, Hong Kong SAR

Acknowledgments

The authors thank the Research Impact Fund, Research Grants Council, University Grants Committee, Hong Kong SAR, for their support.

ORCID iDs

Winnie WT Lam

Kenneth NK Fong

Chi-Wen Chien

Ethical considerations and consent to participate

Experimental procedures were approved by the Human Subjects Research Ethics Committee of the Hong Kong Polytechnic University (Reference No.: HSEARS20230214010). The work was carried out in accordance with The Code of Ethics of the World Medical Association (Declaration of Helsinki). After the nature, purpose, risks and benefits of the study were explained, participants gave written informed consent to participate in the study.

Consent for publication

All authors have approved this manuscript for publication. It has not been previously published, nor is it pending publication elsewhere.

Author contributions

The authors confirm that all authors have contributed to the work presented in this manuscript and agree with the following contributorship statement:

Winnie W. T. Lam: Conceptualized the study, designed the methodology, writing of the manuscript, data collection. Contributed to data analysis and interpretation.

Kenneth N. K. Fong: Supervised the project, conceptualized the study, designed the methodology. Drafting the manuscript and reviewed the final version. Reviewed and edited the manuscript for intellectual content.

Chi-Wen Chien: Conducted the literature review, contributed to study design, and performed statistical analysis. Participated in revising the manuscript.

All authors have approved the final version of the manuscript and agree to be accountable for all aspects of the work.

Funding

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This research project was partially funded by Research Impact Fund (Ref. no.: R5028-20F), Research Grants Council, University Grants Committee, Hong Kong SAR, to KNKF, and Research Centre for Assistive Technology, The Hong Kong Polytechnic University (Ref. no.: CE0E) to KNKF.

Declaration of conflicting interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Data availability statement

The data that support the findings of this study are available on request from the corresponding author [Kenneth NK Fong]. The data are not publicly available due to ethical restrictions, their containing information that could compromise the privacy of research participants.

Appendix 1. Details of the statistical analysis

Kinematic data, including completion time, angular waveforms, and joint angles at target positions, were extracted from the MMC system. The first trial in each task was a practice trial and was excluded from analysis. Averages from the second to fifth trials were used for statistical analysis. Comparisons of the joint angles in the target positions (ATP) were carried out using independent t-tests between: (a) the affected side and the unaffected side in the stroke population; and (b) the affected side in the stroke population and the dominant side in their healthy counterparts. Comparisons of the completion times in Task 7 and the ATP in Tasks 1 to 6 were carried out between the affected side in the stroke population with the higher functional level (FTHUE level five or above), the stroke population with the lower functional level (FTHUE level four or below), and the dominant side of the healthy subjects, using an ANOVA with post hoc comparison. Differences in the angular waveforms between the affected hand and the unaffected hand in the stroke population, and the two sides of the healthy subjects, were compared using the coefficient of multiple correlation (CMC) and the root mean square error (RMSE). Correlations between the assessment results corresponding to the actions and the completion times for Task 7, the ATP, and the CMC values were evaluated using Pearson's r correlation. The linear support vector machine (SVM) used a linear kernel. l₂ regularization was implemented using a logistic regression model. The cutoff scores for each class were chosen by considering the cutoff that maximizes the F1 score balancing precision and recall.

References

Jung

H-Y

. Rehabilitation in subacute and chronic stage after stroke. In: Lee

(ed.) Stroke revisited: diagnosis and treatment of ischemic stroke. Berlin, Germany: Springer, 2017, pp.351–360.

Poole

Whitney

. Assessments of motor function post stroke: a review. Phys Occup Ther Geriatr 2001; 19: 1–22.

Kim

Parnandi

Eva

, et al. The use of wearable sensors to assess and treat the upper extremity after stroke: a scoping review. Disabil Rehabil 2022; 44: 6119–6138.

Mündermann

Corazza

Andriacchi

. The evolution of methods for the capture of human movement leading to markerless motion capture for biomechanical applications. J Neuroeng Rehabil 2006; 3: 1–11.

Wade

Needham

McGuigan

, et al. Applications and limitations of current markerless motion capture methods for clinical gait biomechanics. Peer J 2022; 10: e12995.

Scataglini

Abts

Van Bocxlaer

, et al. Accuracy, validity, and reliability of markerless camera-based 3D motion capture systems versus marker-based 3D motion capture systems in gait analysis: a systematic review and meta-analysis. Sensors 2024; 24: 3686.

Francia

Motta

Donno

, et al. Validation of a MediaPipe system for Markerless motion analysis during virtual reality rehabilitation. In: Gamberini

Carta

Spoto

, et al. (eds) Extended reality. Cham: Springer Nature Switzerland, 2024, pp.123–135.

Eichler

Hel-Or

Shimshoni

, et al. 3D motion capture system for assessing patient motion during fugl-Meyer stroke rehabilitation testing. IET Comput Vision 2018; 12: 963–975.

Fong

Kwan

RYC

. Telerehabilitation (remote therapy). In: Dannefer

Phillipson

(eds) Encyclopedia of gerontology and population aging. Cham, Switzerland: Springer Nature Switzerland AG, 2020, pp.1–7.

10.

Lorenz

Skjæret-Maroni

. A review of combined functional neuroimaging and motion capture for motor rehabilitation. J Neuroeng Rehabil 2024; 21: 1–29.

11.

Lam

Fong

. The application of markerless motion capture (MMC) technology in rehabilitation programs: a systematic review and meta-analysis. Virtual Real 2023; 27: 3363–3378.

12.

Sohn

Sipahi

Sanger

, et al. Portable motion-analysis device for upper-limb research, assessment, and rehabilitation in non-laboratory settings. IEEE J Transl Eng Health Med 2019; 7: 1–14.

13.

Lam

Tang

Fong

. A systematic review of the applications of markerless motion capture (MMC) technology for clinical measurement in rehabilitation. J Neuroeng Rehabil 2023; 20: 57.

14.

Lam

Fong

. Validity and reliability of upper limb kinematic assessment using a markerless motion capture (MMC) system: a pilot study. Arch Phys Med Rehabil 2024; 105: 673–81. e2.

15.

Fong

Chan

, et al. Development of the Hong Kong version of the functional test for the hemiplegic upper extremity (FTHUE-HK). Hong Kong J Occup Ther 2004; 14: 21–29.

16.

Woodbury

Velozo

Richards

, et al. Dimensionality and construct validity of the Fugl-Meyer assessment of the upper extremity. Arch Phys Med Rehabil 2007; 88: 715–723.

17.

Morris

Uswatte

Crago

, et al. The reliability of the wolf motor function test for assessing upper extremity function after stroke. Arch Phys Med Rehabil 2001; 82: 750–755.

18.

Kang

. Sample size determination and power analysis using the G* Power software. J Educ Eval Health Prof 2021; 18: 17.

19.

Fong

Yoyo

, et al. Effects of sensory cueing on voluntary arm use for patients with chronic stroke: a preliminary study. Arch Phys Med Rehabil 2011; 92: 15–23.

20.

Hoonhorst

Nijland

Van Den Berg

, et al.

How do Fugl–Meyer arm motor scores relate to dexterity according to the action research arm test at 6 months poststroke?

Arch Phys Med Rehabil 2015; 96: 1845–1849.

21.

Xia

. Prism vote: a stratified statistical framework to perform prediction for complex diseases. Hong Kong: The Chinese University of Hong Kong, 2020.

22.

Gray

Rice

Garland

. Factors that influence muscle weakness following stroke and their clinical implications: a critical review. Physiother Canada 2012; 64: 415–426.

23.

Fong

Ting

Zhang

, et al. The effect of mirror visual feedback on spatial neglect for patients after stroke: a preliminary randomized controlled trial. Brain Sci 2022; 13: 3.

24.

Zamin

Tang

Stevens

, et al. Abnormal motION capture in aCute stroke (BIONICS): a low-cost tele-evaluation tool for automated assessment of upper extremity function in stroke patients. Neurorehabil Neural Repair 2023; 37: 591–602.

25.

Abdullah

Sofyan

. Machine learning in safety and health research: a scientometric analysis. Int J Inform Sci Manag (IJISM) 2023; 21: 17–37.

26.

Ceriola

Taborri

Donati

, et al. Comparative analysis of markerless motion capture systems for measuring human kinematics. IEEE Sensors J 2024; 24: 28135–28144.

27.

Bonnechère

Sholukha

Omelina

, et al. 3D Analysis of upper limbs motion during rehabilitation exercises using the KinectTM sensor: development, laboratory validation and clinical application. Sensors (Basel, Switzerland) 2018; 18: 2216.

28.

Knippenberg

Verbrugghe

Lamers

, et al. Markerless motion capture systems as training device in neurological rehabilitation: a systematic review of their use, application, target population and efficacy. J Neuroeng Rehabil 2017; 14: 1–11.