Sage Journals: Discover world-class research

Abstract

Background

Telemedicine support virtual consultations and evaluations in hand surgery for patients in remote areas during the COVID-19 era. However, traditional physical examination is challenging in telemedicine and it is inconvenient to manually measure the hand range of motion (ROM) from images or videos. Here, we propose an automatic method using the hand pose estimation technique, aiming to measure the hand ROM from smartphone images.

Methods

Twenty-eight healthy volunteers participated in the study. An eight-hand gestures measurement protocol and the Google MediaPipe Hands were used to analyze images and calculate the ROM automatically. Manual goniometry was also performed according to the guideline of the American Medical Association. The correlation between the automatic and manual methods was analyzed by the intraclass correlation coefficient and Pearson correlation coefficient. The clinical acceptance was testified using Bland-Altman plots.

Results

A total of 32 parameters of each hand were measured by both methods, and 1792 measurement results were compared. The mean difference between automatic and manual methods is −2.21 ± 9.29° in the angle measurement and 0.48 ± 0.48 cm in the distance measurement. The intraclass correlation coefficient of 75% of parameters was higher than 0.75, the Pearson correlation coefficient of 84% of parameters was over 0.6, and 40.6% of parameters reached well-accepted clinical agreements.

Conclusions

The proposed method provides a helpful protocol for automatic hand ROM measurement based on smartphone images and the MediaPipe Hands pose estimation technique. The automatic measurement is acceptable and comparable with existing methods, showing a possible application in the telemedicine examination of hand surgery.

Keywords

Hand measurement range of motion hand pose estimation telemedicine smartphone

Introduction

The range of motion (ROM) is fundamental in the physical examination and the follow-ups in clinical practice.^1,2 For hand surgeons and therapists, measurement of hand ROM is an objective and quantified way to assess the retained or improved motor function of digits, and provide evidence for estimation of the outcome or the progress of surgery or rehabilitation therapy.³ Traditionally, manual goniometry has been the widely accepted method for ROM measuring in clinical practice,⁴ which could be traced back to the early 1900s.⁵ However, the dependence on experienced raters, face-to-face measurement, and physical contact brings difficulties, especially in COVID-19 and beyond.

At present, smartphones have become valuable and acceptable tools in telemedicine.⁶ It allows patients to interact and consult with experts more easily through images or videos, significantly reducing the travel burden for patients in remote or pandemic lockdown areas. Additionally, in hand and upper extremity surgery, previous studies have reported the utility and benefit of telemedicine for diagnosis and postoperative follow-up.^7,8 Patients also show high interest and preferences for virtual visits.⁹ However, it has been concerned that virtual consultation only permits primary interaction between clinicians and patients without physical contact. It hence highly restricts the application of physical examination in traditional ways. An available solution is visually estimating the ROM by giving instructions and asking the patient to demonstrate their hand movement. Despite visual assessments showing excellent agreement with conventional measurements for the elbow and wrist joints,^10,11 how to measure the complex movement of digit joints via images or videos remains difficult for researchers.^12–14 The other solution is photographic goniometry, which manually measures the joint angles from images. This method has been investigated in healthy volunteers and patients with fractures of the phalanges, tendon rupture repair, and Dupuytren disease with good agreements.^15–18 However, both methods are still subjective and time-consuming to obtain the data. A more convenient and automatic technique would be valuable in telemedicine examination, particularly for analyzing the function of digits in hand surgery.

With continuous development in computer vision, the hand pose estimation technique offers a contactless solution to predict the articulated joint locations from images or videos for hand motion analysis.^19–21 Commonly, hand pose estimation is based on images with depth information or just ordinary RGB images. The former requires depth camera devices, like Microsoft Kinect,²² Intel RealSense,²³ and Leap Motion,¹⁸ and have been evaluated in home care and telerehabilitation of the upper limb.^24–26 However, these devices may not be available for every patient. The other method could extract key points from a single image via general cameras or smartphones. The MediaPipe Hands is a deep learning-based hand-tracking solution with only RGB input, and it can predict 21 key points coordinates of a hand.²⁷ It is one of the state-of-the-art algorithms trained on some of the largest and most diverse datasets from Google.²⁸ Like other pose estimation methods, the MediaPipe Hands are popular in gesture recognition and control for human–computer interaction.^27–29 However, it remains unknown whether it would be helpful in measuring hand ROM for medical purposes.

We aim to propose an automatic hand ROM measurement method based on MediaPipe Hands and test whether it would achieve an acceptable agreement for telemedicine applications via a simple smartphone device.

Materials and methods

According to the pilot study, we calculated the sample size of 26:26 for an equivalent test ( $α = 0.05$ , $β = 0.2$ , equivalence level = ± 5, mean 4.3, standard deviation = 1.0). The inclusion criteria for participants were: wellness of both hands without loss of digits, no symptoms or abnormal findings in the physical examination, and age above 18. Participants with any musculoskeletal disorder, pain, or deformity that influenced the hand movement and those who were unwilling to participate were excluded. Between April 2022 and June 2022, 28 Chinese volunteers who lived in Guangzhou were invited to participate in the study. Age, gender, weight, height, and dominant hand were recorded. The participants fully understood the study's protocol and cooperated to perform the requested gestures, with the verbal consent of their hand images used for research purposes.

In this study, we followed the guides for hand impairment evaluation set by Swanson and the American Medical Association (AMA),^30–32 which suggest the required ROM parameters. According to the references, the following movement should be included: (1) The extension/flexion of the metacarpophalangeal (MCP) joints, proximal interphalangeal (PIP) joints and distal interphalangeal (DIP) joints of the index, middle, ring and little finger on the sagittal plane, as well as the MCP joint and interphalangeal (IP) joint of the thumb on the coronal plane; (2) The radial-abduction/ radial-adduction of the thumb carpometacarpal (CMC) joint on the coronal plane. (3) The opposition/adduction movement of the thumb. In total, 32 parameters for ROM measurement were required to be recorded. The measurement parameters and methods are shown in Table 1.

Table 1.

The 32 ROM parameters and the measurement of MediaPipe and manual methods.

No	Parameter	MediaPipe measurement	Manual measurement
Thumb
01	MCP Extension	Angle between 01–02 & 02–03	Angle between the dorsal midline of proximal phalanx & metacarpal.
02	MCP Flexion	Angle between 01–02 & 02–03
03	IP Extension	Angle between 02–03 & 03–04	Angle between the dorsal midline of distal phalanx & proximal phalanx.
04	IP Flexion	Angle between 02–03 & 03–04
05	Radial abduction	Angle between 01–02 & 0–05	Angle between the dorsal midline of the first and second metacarpal.
06	Radial adduction	Angle between 01–02 & 0–05
07	Opposition	Euclidean distance of 03–09	Largest distance between the crease of thumb IP joint and the distal palmar crease over the middle finger MCP joint.
08	Adduction	Euclidean distance of 03–17	Shortest distance between the crease of thumb IP joint and the distal palmar crease over the little finger MCP joint.
Index finger			MCP Extension and Flexion: Angle between the dorsal midline of proximal phalanx & metacarpal; PIP Extension and Flexion: Angle between the dorsal midline of middle phalanx & proximal phalanx; DIP Extension and Flexion: Angle between the dorsal midline of distal phalanx & middle phalanx.
09	MCP Extension	Angle between 05'–06 & 05–06
10	MCP Flexion	Angle between 05'–06 & 05–06
11	PIP Extension	Angle between 05–06 & 06–07
12	PIP Flexion	Angle between 05–06 & 06–07
13	DIP Extension	Angle between 06–07 & 07–08
14	DIP Flexion	Angle between 06–07 & 07–08
Middle finger
15	MCP Extension	Angle between 09'–10 & 09–10
16	MCP Flexion	Angle between 09'–10 & 09–10
17	PIP Extension	Angle between 09–10 & 10–11
18	PIP Flexion	Angle between 09–10 & 10–11
19	DIP Extension	Angle between 10–11 & 11–12
20	DIP Flexion	Angle between 10–11 & 11–12
Ring finger
21	MCP Extension	Angle between 13'–14 & 13–14
22	MCP Flexion	Angle between 13'–14 & 13–14
23	PIP Extension	Angle between 13–14 & 14–15
24	PIP Flexion	Angle between 13–14 & 14–15
25	DIP Extension	Angle between 14–15 & 15–16
26	DIP Flexion	Angle between 14–15 & 15–16
Little finger
27	MCP Extension	Angle between 17'–18 & 17–18
28	MCP Flexion	Angle between 17'–18 & 17–18
29	PIP Extension	Angle between 17–18 & 18–19
30	PIP Flexion	Angle between 17–18 & 18–19
31	DIP Extension	Angle between 18–19 & 19–20
32	DIP Flexion	Angle between 18–19 & 19–20

The key point number of MediaPipe measurement is demonstrated in Figure 2B.

For automatic measurement, we proposed a standard protocol based on eight hand gestures as instructions to guide participants in demonstrating the required ROM movements independently (Figure 1). We used a smartphone (iPhone XS Max, Apple Inc., image resolution: 1980*1080p) to take the images and mimic the setting of telemedicine visits. All images were taken from the palmar view for better key points detection and prediction. For manual measurement, we followed the AMA guideline and measured the ROM with a universal goniometer.³² Measurement was performed by two hand surgeons in our department. For both methods, the zero position was defined as the neutral position of a fully extended joint.³³ All data were expressed as positive values, except hyperextension was recorded in negative values.

Figure 1.

The eight-hand-gesture automatic measurement protocol with examples.

A simple experimental setup was prepared for automatic measurement (Figure 2A). The participant was asked to sit by the test table and place a forearm and wrist on a holding fixture (16.5*21.5*17.5cm) to reduce fatigue and shaking of the upper limb. The transverse crease of the wrist was aligned to the edge of the fixture to standardize the hand's position. The smartphone was installed 40 cm above the fixture, and the palm was kept in the center of the image. The forearm was kept in a supination position, and the palm was parallel to the table. A laptop was set in front of the participant to provide visual instructions of the targeted gesture and display the performing gesture in real-time.

Figure 2.

The illustration of the automatic measurement. (A) The setup of the automatic measurement; (B) The 21 key points of the MediaPipe Hands model. The red arrow shows the measuring scale for distance calculation.

Before the experiment, participants were requested to wash and clean their hands. Jewellery, watches, and clothing on the wrist or hand were removed during the investigation. Then, participants were asked to perform the required gestures in order and hold the testing gesture for about 3 seconds. A hand surgeon took three images continuously for each gesture once the participant indicated the gesture had been accomplished thoroughly. A total of 48 images (24 images per hand) of each participant were taken. Breaks were allowed if the participants felt their hands were uncomfortable. The time of image acquisition was recorded. There was a five-minute interval after image acquisition. Then another hand surgeon performed the manual goniometry and recorded the parameters. The time taken for manual measurement was also recorded.

The images were analyzed using a Python-based (Version 3.8) procedure. Briefly, we obtained the 3D coordinates of the 21 key points (Figure 2B) from each image using MediaPipe Hands (Version 0.8.9) and then calculated the parameters using the following equations:

θ = \cos^{- 1} (\frac{\vec{S_{1}} \cdot \vec{S_{2}}}{| \vec{S_{1}} | \cdot | \vec{S_{2}} |})

(1)

d_{p i x} = \sqrt{{(x_{1} - x_{2})}^{2} + {(y_{1} - y_{2})}^{2} + {(z_{1} - z_{2})}^{2}}

(2)

d_{r e a l 1} = d_{p i x 1} * \frac{d_{r e a l 2}}{d_{p i x 2}}

(3)

For angle parameters calculation, we employed a similar method used by Zhu et al. as Equation (1).²² We modified the angle calculation of the MCP joints (Figure 3) to decrease the angle deviation on the coronal plane in the extension/flexion movement of fingers (there was only one key point of the wrist in the MediaPipe Hands model). The distance parameters were calculated in 2 steps, much the same as Pathi et al. did.³⁴ We set a measurement scale on the fixture (Figure 2B) to obtain the ratio of pixel and real-world distance and then automatically calculated the distance using Equations (2) and (3). We used the mean value of the three corresponding images as the final result for each parameter. Equation (1): Angle calculation. θ represents the target angle,

\vec{S_{1}}

and

\vec{S_{2}}

represent the space vector of related phalangeal or metacarpal bones for angle calculation. Equation (2): pixel distance calculation.

d_{p i x}

represents the pixel distance in the image.

x_{1}, y_{1}, z_{1}

and

x_{2}, y_{2}, z_{2}

represent the 3D coordinates of the two points for pixel distance calculation. Equation (3): real-world distance calculation.

d_{p i x 1}, d_{r e a l 1}

represent the distance in pixel and real-world of the target distance.

d_{p i x 2}, d_{r e a l 2}

represent the distance in pixel and real-world of the measuring scale.

Figure 3.

The diagram of modified MCP joint angle calculation. The palm plane is determined by two MCP points and the wrist point. PIP’ is the projection point of PIP on the palm plane. θ represents the modified MCP joint angle.

Measurement results were expressed by the mean and standard deviation (Mean ± SD). The reliability of the automatic methods was analyzed in the following steps: (1) The deviation between the two measurement methods was assessed for each parameter. Then the 95% confidence interval (CI) was calculated and compared with the range of clinical acceptance. The acceptance for angle parameters was defined as ± 5° since it was acceptable in clinical practice.^26,35 Acceptance for distance parameters was defined as ± 1 cm based on the impairment algorithm proposed by Swanson.³¹ (2) The correlation of the two methods was assessed by intraclass correlation coefficient (ICC) and Pearson correlation coefficient. ICC value explanation was defined as follows: 0–0.20: unacceptable, 0.20–0.40: questionable, 0.41–0.60: good, 0.61–0.80 very good, 0.81–1.00: excellent. In the Pearson correlation coefficient, 1 indicates a total positive linear correlation, 0 means no linear correlation, and −1 shows a total negative linear correlation. (3) The clinical agreement of the two methods was assessed by Bland-Altman plots to visualize the data and compare the limits of agreement (LOA).³⁶ In this study, a well-acceptable clinical agreement was defined as the LOA was entirely within the range of clinical acceptance. As the key point coordinates were constant for the same image using MediaPipe Hands, the experiment for test–retest was unnecessary. Statistical analysis was performed by IBM SPSS v25 (IBM, Armonk, NY, USA).

Results

In the study, 56 hands of 28 volunteers were measured. The mean age of volunteers was 26.9 ± 2.9 years, the female-to-male ratio was 1:3, and the BMI was 23.2 ± 3.1 (Table 2). All participants were right-handed and completed both measurement methods. The mean completion time for image taking and manual measurement were 1.5 ± 0.4 min and 11.8 ± 1.0 min. All images were successfully detected and analyzed by MediaPipe Hands. There was no missing data for all 1792 (56 × 32) parameters.

Table 2.

Demographics of participants.

No.	Gender	Age (Y)	BMI	Dominant hand
P1	Male	25	24.73	Right
P2	Male	32	26.45	Right
P3	Male	29	27.10	Right
P4	Male	28	26.12	Right
P5	Male	28	29.07	Right
P6	Male	26	17.84	Right
P7	Male	25	26.67	Right
P8	Male	30	24.90	Right
P9	Female	28	19.27	Right
P10	Male	28	25.20	Right
P11	Male	23	22.86	Right
P12	Male	24	23.04	Right
P13	Male	28	21.31	Right
P14	Male	25	21.15	Right
P15	Female	24	19.81	Right
P16	Female	26	24.54	Right
P17	Female	28	17.58	Right
P18	Female	32	21.99	Right
P19	Male	25	26.40	Right
P20	Male	25	32.74	Right
P21	Female	26	21.63	Right
P22	Male	24	20.99	Right
P23	Male	25	22.41	Right
P24	Male	24	21.14	Right
P25	Female	27	17.36	Right
P26	Female	29	17.90	Right
P27	Male	25	26.03	Right
P28	Male	35	22.86	Right

The results of the automatic and manual measurements are summarized in Table 3. The mean difference is −2.21 ± 9.29° for all angle parameters and 0.48 ± 0.48 cm for distance parameters, respectively. According to the 95% CI, 81.25% of the parameters had a deviation less than ± 5° or ± 1 cm, and both were considered acceptable. The extensions of all digits, the thumb flexion, and the thumb radial-adduction achieved the best results, with deviations of less than 2°. The maximum difference was found in the flexion of the index MCP joint (mean: −25.01°, 95% CI: −27.485 to −22.531). Similarly, differences were also evident in the flexion of the middle, ring, and little finger MCP joints (mean, 14.33°, 15.23°, 11.64°).

Table 3.

Results of two ROM measurement methods.

Digit	Parameter	MediaPipe (Mean ± SD)	Goniometry (Mean ± SD)	Mean error	95% CI of mean error	Mean absolute error
Thumb	MCP_E	−5.81 ± 6.58	−6.14 ± 5.66	0.33	(−0.305,0.973)	2.02
	MCP_F	64.74 ± 8.51	65.21 ± 7.1	−0.48	(−1.107,0.154)	1.94
	IP_E	−22.08 ± 7.2	−22.16 ± 6.3	0.08	(−0.584,0.741)	1.92
	IP_F	60.69 ± 9.64	60.66 ± 8.52	0.03	(−0.635,0.701)	1.99
	radial_ad	15.16 ± 3.88	15.36 ± 2.39	−0.20	(−0.748,0.349)	1.64
	radial_ab	31.21 ± 5.65	34.52 ± 5.37	−3.31	(−3.532,−3.088)	3.31
	opposition	7.74 ± 0.98	7.21 ± 0.53	0.53	(0.366,0.694)	0.64
	adduction	3.13 ± 0.51	2.71 ± 0.29	0.42	(0.344,0.499)	0.42
Index	MCP_E	2.98 ± 3.87	2.16 ± 2.59	0.82	(0.250,1.384)	1.96
	MCP_F	64.47 ± 9.61	89.48 ± 2.37	−25.01	(−27.485,−22.531)	25.11
	PIP_E	5.55 ± 3.4	2.7 ± 2.24	2.85	(2.429,3.269)	2.93
	PIP_F	93.47 ± 9.15	100.54 ± 4.41	−7.07	(−9.210,−4.928)	7.85
	DIP_E	−0.4 ± 4.25	−0.48 ± 2.71	0.08	(−0.569,0.737)	2.06
	DIP_F	76.36 ± 8.14	77.36 ± 5.84	−1.00	(−2.042,0.048)	2.92
Middle	MCP_E	−3.11 ± 5.4	−3.66 ± 4.16	0.56	(−0.008,1.118)	1.83
	MCP_F	75.88 ± 12.42	90.21 ± 2.92	−14.33	(−17.471,−11.198)	15.58
	PIP_E	1.02 ± 6.45	0.05 ± 4.85	0.97	(0.402,1.533)	1.97
	PIP_F	94.33 ± 8.06	99.98 ± 3.6	−5.65	(−7.577,−3.730)	6.25
	DIP_E	−1.91 ± 3.57	−1.79 ± 2.52	−0.12	(−0.607,0.357)	1.50
	DIP_F	74.61 ± 10.31	76.13 ± 6.64	−1.52	(−2.816,−0.222)	3.72
Ring	MCP_E	0.93 ± 4.9	−0.63 ± 4.14	1.56	(1.009,2.105)	2.16
	MCP_F	74.99 ± 12.77	90.21 ± 3.38	−15.23	(−18.376,−12.08)	17.21
	PIP_E	1.03 ± 6.87	0.14 ± 4.86	0.89	(0.214,1.569)	2.27
	PIP_F	106.72 ± 7.65	103.77 ± 3.48	2.96	(1.386,4.526)	5.18
	DIP_E	−2.13 ± 4.1	−1.68 ± 2.94	−0.45	(−1.032,0.124)	1.76
	DIP_F	81.2 ± 11.73	78.77 ± 7.42	2.43	(0.690,4.174)	4.75
Little	MCP_E	2.51 ± 5.28	1.3 ± 4.28	1.21	(0.735,1.685)	1.81
	MCP_F	78.68 ± 26.75	90.32 ± 3.95	−11.64	(−18.137,−5.139)	24.29
	PIP_E	−4.68 ± 3.57	−4.21 ± 2.68	−0.47	(−1.079,0.148)	1.83
	PIP_F	107.07 ± 11.61	104.21 ± 3.66	2.85	(0.354,5.351)	7.27
	DIP_E	−7.67 ± 3.86	−5.91 ± 2.34	−1.75	(−2.296,−1.213)	2.09
	DIP_F	85.49 ± 13.13	81.21 ± 7.98	4.28	(2.318,6.240)	6.08

*The opposition and adduction are recorded in centimeters, and the rest parameters are recorded in degrees.

ROM: range of motion; SD: standard deviation; CI: confidence interval; _E: extension; _F: flexion; _ab: abduction; _ad: adduction.

In ICC and the Pearson correlation coefficient (Table 4), 75% of parameters (24/32) achieved excellent or very good correlation, except the MCP joint flexion of the four fingers, which was regarded as questionable correlation (ICC: 0.126, 0.157, 0.209, 0.195). 84% of parameters (27/32) resulted in a Pearson correlation coefficient over 0.6, showing a strong positive correlation.

Table 4.

Summary of the correlation between the two measurement methods.

Digit	Parameters	Pearson	ICC
Thumb	MCP_E	0.892 **	0.881
	MCP_F	0.971 **	0.955
	IP_E	0.926 **	0.91
	IP_F	0.97 **	0.962
	radial_ad	0.895 **	0.798
	radial_ab	0.99 **	0.989
	opposition	0.84 **	0.699
	adduction	0.937 **	0.834
Index	MCP_E	0.786 **	0.713
	MCP_F	0.271*	0.126
	PIP_E	0.851 **	0.801
	PIP_F	0.486**	0.38
	DIP_E	0.832 **	0.749
	DIP_F	0.895 **	0.848
Middle	MCP_E	0.936 **	0.905
	MCP_F	0.352**	0.157
	PIP_E	0.962 **	0.91
	PIP_F	0.455**	0.338
	DIP_E	0.881 **	0.83
	DIP_F	0.927 **	0.844
Ring	MCP_E	0.911 **	0.898
	MCP_F	0.422**	0.209
	PIP_E	0.965 **	0.91
	PIP_F	0.681 **	0.514
	DIP_E	0.863 **	0.817
	DIP_F	0.864 **	0.78
Little	MCP_E	0.917 **	0.893
	MCP_F	0.674 **	0.195
	PIP_E	0.767 **	0.736
	PIP_F	0.719 **	0.413
	DIP_E	0.85 **	0.723
	DIP_F	0.871 **	0.773

^**

Correlation is significant at the 0.01 level (two-tailed).

Correlation is significant at the 0.05 level (two-tailed).

ICC: intraclass correlation coefficient; _E: extension; _F: flexion; _ab: abduction; _ad: adduction.

The clinical agreement was analyzed using Bland–Altman plots (Figure 4). In general, 40.6% of parameters (13/32) achieved a well-acceptable clinical agreement when comparing the LOA and the clinical acceptance. Some parameters like thumb MCP flexion, index finger PIP extension, middle finger PIP extension, ring finger MCP and PIP extension, and little finger DIP extension nearly reached clinical acceptance. The rest 13 parameters were considered to have failed in the clinical agreement.

Figure 4.

Bland–Altman plots for analyzing clinical agreement. There are 56 dots for every parameter. In each plot, the x-axis represents the mean and the y-axis represents the difference between automatic and manual methods. The bold black line represents the mean difference, the dotted lines represent the limit of agreement, and the light red area represents the range of clinical acceptance (±5° or ± 1 cm).

Discussion

Hand ROM measurement is a critical component in the examination of motor function, disease diagnosis, and impairment evaluation.³⁷ Unfortunately, acquiring the parameters in the telemedicine visit was inconvenient. Our goal was to provide a new method that uses smartphones and images to automatically obtain and calculate hand ROM, trying to alleviate the manual measurement burden and achieve acceptable clinical agreement.

This study provided an eight-gesture-based measurement protocol to guide users in taking the required images with a smartphone. Compared with other related works,^26,38,39 the study offered a protocol with the comprehensive movement of the thumb in gestures 2–6 and separate digit flexion movement into gestures 7 and 8. In our study, we noticed a measurement improvement in DIP flexion that difference with the manual measurement was only from −1.75° to 0.08°. In the study of Nizamis et al.,³⁸ all digits were flexed together (like gesture 8), and only MCP flexion of the index finger achieved satisfactory agreement. Pham et al. also used a “fist position” for measuring the flexion of fingers using a Creative Senz3D camera.³⁹ They found that DIP had a more noticeable difference and variance than the other joints. We believe the improvement was caused by gesture 7, in which the key points of the fingertip, DIP and PIP joints were not severely obstructed and led to a better coordinate prediction of MediaPipe Hands. Therefore, the angle of DIP joints could achieve a more reasonable result. However, the unsatisfied PIP and MCP flexion measurement had not been solved. Our contribution was providing a clinical guideline-based ROM measurement protocol for hand pose estimation and automatic ROM measurement. It contained 32 indispensable parameters for hand functional evaluation, which were also recommended in telemedicine examination.¹²

The reliability of automatic methods was assessed by comparing the result with manual measurement. Although the quality of manual measurement relied on the rater's experience, it had been a cheap, simple, and well-accepted method in clinical practice. It was an essential benchmark for clinical acceptance and was more well-known among hand surgeons. In traditional goniometry, Ellis et al. reported an inter-rater variance of 7° to 9° and 0.7 to 0.9 cm between therapists, with an intra-rater variance of 4° to 5° and 0.5 to 0.6 cm for repeated measurement.⁴⁰ In manual photographic measurement, Georgeu et al. compared lateral image measurement of the ring or little fingers with standard goniometry in 20 patients.⁴¹ In their study, the average difference was 1° (SD: 6°, 95% Cl: 0° to + 2°), and the 95% LOA was −11° to + 13°, which was beyond the range of clinical acceptance. Meals et al. found that photo-based measurements were reliable within and between observers, but only 27.3% (3/11) of photo-based measurements were in acceptable agreement (r≥0.70) with manual goniometry.⁴² We found our results comparable to these measurements. In our study, even if we set a ± 5° as the level of clinical acceptance, 40.6% of parameters had strictly reached the clinical agreement, and another 18.8% of parameters had almost achieved the agreement. We have also noticed that the key point coordinates remain unchanged using MediaPipe Hands for the same image, which means our method has perfect repeatability. It also indicated that the automatic method could assist less experienced examiners in the challenging visual measurement of the hand, promote intra-rater reliability and facilitate the process of tele-examination.

There has been an increasing interest in hand pose estimation techniques for medical research and application. For example, Leap Motion has been used in hand rehabilitation assessments with 3D skeletal data from RGB-D images.^43,44 Wang et al. developed a Leap Motion-based virtual reality system to improve the motor function of subacute stroke patients.⁴⁵ Butt et al. testified an automatic classification of Parkinson's disease by assessing the movement of the hand and forearm.⁴⁶ Martini et al. used OpenPose software to evaluate the Hand-To-Nose test for stroke patients.²¹ Their works focused more attention on the kinematics feature of the upper limb, and the implementation was complicated. In the research field of automatic hand ROM measurement, the clinical agreement was unfavorable in related studies.^38,47–49 The reason might be the lack of large datasets for hand pose estimation compared with face or body,²⁰ and the occlusion of the hand is still a challenging task for manual annotation.⁵⁰ Moreover, hand datasets for medical applications or clinical settings did not exist. Further development in the 3D annotation of the hand, more clinical datasets, and more robust algorithms would hopefully offer better key points prediction and improve the automatic ROM parameters calculation.

The proposed method has several advantages: It can reduce the workload for clinicians, and the measurement time was approximately reduced to one-sixth of traditional manual goniometry. Meanwhile, it requires no measuring tools such as tape, caliper, or goniometer, even no necessity of experienced raters. It is simple enough that all images were taken successfully in a short time without supervision. Our method has probably provided a promising solution as a constant, objective, and convenient measurement tool in telemedicine. It can evaluate whether a joint was extended or flexed, even if the digit flexion measurement has not reached clinical acceptance. By qualitatively analyzing the abnormal movement of the particular joint(s), our method is helpful for the primary evaluation or screening of diseases or injuries, such as the radial, ulnar, or median nerve injury,^51–53 tendon rupture, or severity classification of stroke and Parkinson disease.

Our study has several limitations. Firstly, the new method was only testified in healthy and young volunteers who had no obvious ROM limitation. Similarly, whether MediaPipe Hands is sensitive enough to predict key points with limited ROM is uncertain to answer. Secondly, the images were taken in a relatively standardized environment rather than by the subjects themselves at home or other places. Further research should include more complex application environments. Also, the comparatively small number of participants might reduce the robustness and generalizability of the proposed method.

Conclusion

The measurement of hand range of motion is challenging in telemedicine examination. This study demonstrates a new thought for automatic measurement using RGB images with MediaPipe Hand pose estimation via a smartphone. A comprehensive and clinical-based protocol with eight hand gestures was provided to bridge the traditional manual and automatic measurements. Our method obtained good clinical agreements and is suitable for other hand pose estimation studies. It indicates excellent potential for telemedicine application with the benefits of convenience, acceptable reliability, and simple smartphone implementation.

Ethics approval and informed consent

Ethical approval to report this case was obtained from the institutional review board of The First Affiliated Hospital of Sun Yat-sen University (ID: [2021]387).

Verbal informed consent was obtained from the patient(s) for their anonymized information to be published in this article.

Footnotes

Acknowledgments

The authors thank Mr Bai Leng, Mr Zhenguo Lao, Ms. Jianwen Zheng, and Guangzhou HuanTek Co., Ltd for technical help, writing assistance, and general support.

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

Authorship confirmation

Fanbin Gu: Investigation (equal), Software (equal), Visualization (equal), Writing – original draft (lead). Jingyuan Fan: Investigation (equal), Software (equal), Visualization (equal), Formal analysis (lead). Zhaoyang Wang: Resources (lead), Data Curation (lead). Xiaolin Liu: Conceptualization (equal). Jiantao Yang: Methodology (equal), Project administration (lead), Writing – review & editing (lead). Qingtang Zhu: Conceptualization (equal), Methodology (equal), Supervision (lead).

We have confirmed that all the authors meet the authorship criteria.

ORCID iD

Fanbin Gu

Author biographies

Fanbin Gu is a PhD student in Clinical Medicine. His research interest is artificial intelligence for hand surgery.

Jingyuan Fan is a PhD student in Clinical Medicine. His research interest is artificial intelligence for hand surgery.

Zhaoyang Wang is a PhD student in Clinical Medicine. His research interest is artificial intelligence for hand surgery.

Xiaolin Liu is a professor at the Department of Microsurgery, Orthopaedic Trauma and Hand Surgery, The First Affiliated Hospital, Sun Yat-sen University. His research interest is peripheral nerve repair and biofabrication of nerve substitutes.

Jiantao Yang is an associate professor at the Department of Microsurgery, Orthopaedic Trauma and Hand Surgery, The First Affiliated Hospital, Sun Yat-sen University. His research interest is peripheral nerve repair and biofabrication of nerve substitutes.

Qingtang Zhu is a professor at the Department of Microsurgery, Orthopaedic Trauma and Hand Surgery, The First Affiliated Hospital, Sun Yat-sen University. He majors in reconstructive microsurgery for extremity and is interested in management of severed extremity injury, peripheral nerve repair, biofabrication of nerve substitutes, and artificial intelligence for telemedicine.

References

Magno

Fleman

Halliburton

, et al. Usefulness of digital measurements for functional evaluation of paediatric elbow range of motion. J Telemed Telecare 2021; 192: 1357633X211001731. doi:10.1177/1357633X211001731

Chang

Lee

, et al. Correlation of return to work outcomes and hand impairment measures among workers with traumatic hand injury. J Occup Rehabil 2011; 21: 9–16.

Young

Daya

Govender

. Functional outcome using early controlled active motion in rehabilitation of a replanted hand: a case report. J Hand Ther 2020; 33: 426–434.

Pratt

Burr

Stott

. An investigation into the degree of precision achieved by a team of hand therapists and surgeons using hand goniometry with a standardised protocol. Hand Therapy 2004; 9: 116–121.

Moore

. The measurement of joint motion; introductory review of the literature. Phys Ther Rev 1949; 29: 195–205.

Hsieh

Yun

Bhatia

, et al. Patient perception on the usage of smartphones for medical photography and for reference in dermatology. Dermatol Surg 2015; 41: 149–154.

Moses

Buchalter

Azad

, et al. Telemedicine during the COVID-19 pandemic: a hand surgery perspective. J Hand Surg Asian Pac Vol 2021; 26: 705–715.

Tofte

Anthony

Polgreen

, et al. Postoperative care via smartphone following carpal tunnel release. J Telemed Telecare 2020; 26: 223–231.

Benavent

Chruscielski

Janssen

, et al. Patient perception and preferences for virtual telemedicine visits for hand and upper extremity surgery. Telemed J E Health 2022; 28: 509–516.

10.

Wagner

Conti Mica

Shin

. Smartphone photography utilized to measure wrist range of motion. J Hand Surg Eur Vol 2018; 43: 187–192.

11.

Meislin

Wagner

Shin

. A comparison of elbow range of motion measurements: smartphone-based digital photography versus goniometric measurements. J Hand Surg Am 2016; 41: 510–515.e1.

12.

Wainberg

Jurisson

Johnson

, et al. The telemedicine hand examination. Am J Phys Med Rehab 2020; 99: 883–883.

13.

Van Nest

Ilyas

Rivlin

. Telemedicine evaluation and techniques in hand surgery. J Hand Surg Glob Online 2020; 2: 240–245.

14.

Welle

Tager

Hackenberg

, et al. Examining the hand in the video consultation. Z Orthop Unfall 2021; 159: 202–208. Die Untersuchung der Hand in der Videosprechstunde

15.

Miyake

Mori

Matsuma

, et al. A new method measurement for finger range of motion using a smartphone. J Plast Surg Hand Surg 2020; 54: 207–214.

16.

Zhao

Blazar

Mora

, et al. Range of motion measurements of the fingers via smartphone photography. Hand (N Y) 2020; 15: 679–685.

17.

Chen

Xian Zhang

Jia Qian

, et al. Measurement of finger joint motion after flexor tendon repair: smartphone photography compared with traditional goniometry. J Hand Surg Eur Vol 2021; 46: 825–829.

18.

Fenelon

Murphy

Galbraith

, et al. Telesurveillance: exploring the use of Mobile phone imaging in the follow-up of orthopedic patients with hand trauma. Telemed E-Health 2019; 25: 1244–1249.

19.

Erol

Bebis

Nicolescu

, et al. Vision-based hand pose estimation: a review. Comput Vis Image Underst 2007; 108: 52–73.

20.

Simon

Joo

Matthews

, et al. Hand Keypoint Detection in Single Images using Multiview Bootstrapping. 30th Ieee Conference on Computer Vision and Pattern Recognition (Cvpr 2017). 2017:4645–4653. https://doi.org/10.1109/Cvpr.2017.494.

21.

Martini

Vale

Boldo

, et al. On the Pose Estimation Software for Measuring Movement Features in the Finger-to-Nose Test 2022 Ieee International Conference on Digital Health (Ieee Icdh 2022). 2022:77–86. https://doi.org/10.1109/Icdh55609.2022.00021.

22.

Zhu

Gan

, et al. A contactless method to measure real-time finger motion using depth-based pose estimation. Comput Biol Med 2021; 131: 104282.

23.

Cejnog

LWX

Cesar

de Campos

, et al. Hand range of motion evaluation for Rheumatoid Arthritis patients. IEEE Int Conf Automa. 2019:569–573.

24.

Zulkarnain

Kim

Adikrishna

, et al. Digital data acquisition of shoulder range of motion and arm motion smoothness using Kinect v2. J Shoulder Elb Surg 2017; 26: 895–901.

25.

Mateo

Carrasco

Aguilar-Rodriguez

, et al. Assessment of Kinect V2 for elbow range of motion estimation in people with haemophilia using an angle correction model. Haemophilia 2019; 25: E165–E173.

26.

Arman

Oktay

Tarakci

, et al. The validity of an objective measurement method using the leap motion controller for fingers wrist, and forearm ranges of motion. Hand Surg Rehabil 2021; 40: 394–399.

27.

Zhang

Bazarevsky

Vakunov

, et al. MediaPipe Hands: On-device Real-time Hand Tracking. 2020.

28.

Adhikary

Talukdar

Sarma

. A Vision-based System for Recognition of Words used in Indian Sign Language Using MediaPipe. Gauhati University, Guwahati, Assam, India; 2021.

29.

Chunduru

Roy

Dasari Romit

, et al. Hand Tracking in 3D Space using MediaPipe and PnP Method for Intuitive Control of Virtual Globe. Amrita Vishwa Vidyapeetham, Amrita School of Engineering, Bengaluru, India; 2021.

30.

Swanson

. Evaluation of impairment of function in the hand. Surg Clin North Am 1964; 44: 925–940.

31.

Swanson

Hagert

Swanson

. Evaluation of impairment of hand function. J Hand Surg Am 1983; 8: 709–722.

32.

Guides to the Evaluation of Permanent Impairment. 2008:634.

33.

Motion. AAoOSCftSoJ. Joint motion : method of measuring and recording. 1966:87.

34.

Pathi

Kiselev

Kristoffersson

, et al. A novel method for estimating distances from a robot to humans using egocentric RGB camera. Sensors (Basel) 2019; 19(14), doi:10.3390/s19143142.

35.

Carter

Pansy

Wolff

, et al. Accuracy and reliability of three different techniques for manual goniometry for wrist motion: a cadaveric study. J Hand Surg Am 2009; 34(8): 1422–1428. doi:10.1016/j.jhsa.2009.06.002.

36.

Bland

Altman

. Statistical methods for assessing agreement between two methods of clinical measurement. Lancet 1986; 1: 307–310.

37.

de los Reyes-Guzman

Dimbwadyo-Terrer

Trincado-Alonso

, et al. Quantitative assessment based on kinematic measures of functional impairments during upper extremity movements: a review. Clin Biomech (Bristol, Avon) 2014; 29: 719–727.

38.

Nizamis

Rijken

NHM

Mendes

, et al. A novel setup and protocol to measure the range of motion of the wrist and the hand. Sensors (Basel) 2018; 18(10): 3230. doi:10.3390/s18103230

39.

Pham

Pathirana

Trinh

Fay

. A Non-Contact Measurement System for the Range of Motion of the Hand. Sensors-Basel 2015; 15: 18315–18333.

40.

Ellis

Bruton

. A study to compare the reliability of composite finger flexion with goniometry for measurement of range of motion in the hand. Clin Rehabil 2002; 16: 562–570.

41.

Georgeu

Mayfield

Logan

. Lateral digital photography with computer-aided goniometry versus standard goniometry for recording finger joint angles. J Hand Surg Br 2002; 27: 184–186.

42.

Meals

Saunders

Desale

, et al. Viability of hand and wrist photogoniometry. Hand (N Y) 2018; 13: 301–304.

43.

Cheng

Yang

, et al. An automatic rehabilitation assessment system for hand function based on leap motion and ensemble learning. Cybern Syst 2021; 52: 3–25.

44.

Guzsvinecz

Szucs

Sik-Lanyi

. Suitability of the Kinect Sensor and Leap Motion Controller-A Literature Review. Sensors (Basel) 2019; 19(5): 1072. doi:10.3390/s19051072

45.

Wang

Xing

, et al. Leap motion-based virtual reality training for improving motor functional recovery of upper limbs and neural reorganization in subacute stroke patients. Neural Regen Res 2017; 12: 1823–1831.

46.

Butt

Rovini

Dolciotti

, et al. Objective and automatic classification of Parkinson disease with leap motion controller. Biomed Eng Online 2018; 17: 168.

47.

Gamboa

Serrato

Castro

, et al. Advantages and limitations of leap motion from a developers’, physical therapists’, and Patients’ perspective. Method Inform Med 2020; 59: 110–116.

48.

Chophuk

Chumpen

Tungjitkusolmun

, et al. Hand Postures for Evaluating Trigger Finger Using Leap Motion Controller. 2015 8th Biomedical Engineering International Conference (Bmeicon). 2015.

49.

Trejo

Vazquez

JPG

Ramirez

MLG

, et al. Hand goniometric measurements using leap motion. Consum Comm Network 2017: 137–141.

50.

Chatzis

Stergioulas

Konstantinidis

, et al. A comprehensive study on deep learning-based 3D hand pose estimation methods. Appl Sci-Basel 2020; 10: 6850.

51.

Soldado

Bertelli

Ghizoni

. High median nerve injury: motor and sensory nerve transfers to restore function. Hand Clin 2016; 32: 209–217.

52.

Bauer

Chaise

. Correction of ulnar claw hand and Wartenberg’s sign. Hand Surg Rehabil 2022; 41S: S118–S127.

53.

Larson

Santosa

Mackinnon

, et al. Median to radial nerve transfer after traumatic radial nerve avulsion in a pediatric patient. J Neurosurg Pediatr 2019; 24: 209–214.

Automatic range of motion measurement via smartphone images for telemedicine examination of the hand

Abstract

Background

Methods

Results

Conclusions

Keywords

Introduction

Materials and methods

Results

Discussion

Conclusion

Ethics approval and informed consent

Footnotes

Acknowledgments

Declaration of conflicting interests

Funding

Authorship confirmation

ORCID iD

Author biographies

References