Sage Journals: Discover world-class research

Abstract

French

Introduction: Hand injuries can lead to lasting impairments that limit one's ability to perform activities of daily living. Many individuals who undergo surgery following a hand injury participate in post-operative rehabilitation. Evidence suggests that geographical and socioeconomic barriers may hinder rehabilitation progress and limit recovery after surgery. Telemedicine and augmented reality may be useful tools for addressing issues concerning accessibility and adherence in rehabilitation. This study validates the use of a novel movement sonification interface that uses Google's MediaPipe hand-tracking technology to map single-digit PIP joint motion to real-time auditory feedback. Methods: Nineteen healthy adults (mean age 32 years; 10 males, 9 females) with no prior exposure to movement sonification participated. Through webcam, participants interacted with a web-based program that mapped fourth digit PIP joint flexion and extension to musical pitch changes (low pitch for extension, high pitch for flexion). Construct, face, content, and predictive criterion validity were assessed through structured surveys after participants viewed and explored the program. Results: Construct validity was 89%, with participants correctly identifying the relationship between fourth digit PIP movement and pitch changes. Both face and content validity were 100%, with participants recognizing the sound-movement relationship and that only the fourth digit produced sound. Predictive criterion validity demonstrated high accuracy, with participants correctly matching hand configurations to sounds (95%) and predicting sounds from muted videos (95%). Conclusions: Results demonstrate excellent construct, face, content, and predictive criterion validity. This supports the feasibility of using music sonification as an intuitive and accessible tool for augmented hand rehabilitation.

Keywords

Movement sonification hand rehabilitation hand therapy augmented reality telemedicine joint motion tracking MediaPipe

Introduction

Hand injuries can lead to lasting impairments that limit one's ability to perform activities of daily living.^1,2 Individuals with hand injuries may experience a myriad of impairments that affect the range of motion, strength, and motor control of their fingers and/or wrist.³ Hand injuries may require surgery for treatment followed by directed and custom hand rehabilitation programs.^4,5

Rehabilitation typically involves working with clinicians through guided exercises aimed at improving range of motion, strength, and motor control.⁶ Participants receive feedback from clinicians that they use to improve their movement patterns, increase strength in the hand, and reduce pain.^7,8 Patients are advised to continue these exercises at home to promote recovery. While current rehabilitation practices benefit many individuals, a sizable proportion do not fully recover and live with chronic functional impairments.⁹ Evidence suggests that geographical and socioeconomic barriers may hinder progress in rehabilitation and limit recovery after surgery.¹⁰ Poor adherence to home-based rehabilitation programing may also be associated with poor outcomes.¹¹ Finally, it has been shown that some patients do not feel motivated to do their home-based exercises because they do not find them engaging or enjoyable.¹² This may stem from a lack of guidance when completing home-based exercises.^13,14 Taken together, several factors seem to impose barriers to progress in post-operative hand rehabilitation.

Telemedicine and augmented reality (AR) may be useful tools for addressing issues concerning accessibility and adherence in rehabilitation. Previous reports have shown that telemedicine and AR approaches can help enhance rehabilitation for pain disorders,^15,16 musculoskeletal disorders,^17,18 cancer,^19–21 stroke,^22–24 and other neurological conditions.^25,26 A novel AR implement that has shown promise is movement sonification in which sounds are linked to movement parameters like displacement, velocity, and acceleration.^27,28 Evidence from stroke rehabilitation has shown that movement sonification may be an effective therapy for improving motor control and restoring function in the hand.²⁹ Importantly, many AR web-based applications are low cost or free-to-use. Thus, AR technology that employs movement sonification may be an engaging and enjoyable means of delivering therapy to patients facing geographic or socioeconomic barriers.^30,31 Movement sonification can also provide objective feedback about movement patterns that guide patients through their home-based rehabilitation exercises potentially improving adherence.²⁸

We have recently developed and submitted for publication our AR web-based interface created using Google's based MediaPipe Hands.³² In summary, specified finger landmarks are detected from a live webcam feed, enabling the real-time calculation of joint angles for joints of each digit (MCPJ, PIPJ, DIPJ) which are then mapped to specific acoustic parameters, producing auditory feedback that dynamically represents joint positions. The purpose of this study was to validate the use of movement sonification application for single-digit movements in healthy adults.

Methods

Subjects

To validate our study, healthy individuals between the ages of 18-80 were asked to partake in our study. All subjects had no prior experience with the movement sonification software. Participants were excluded if they had a previously diagnosed neurological disorder (eg, stroke, Parkinson's disease, cerebellar ataxia), learning disability (eg, ADHD, Dyslexia), or upper extremity injury that limited the motor control of the hand (eg, fracture, tendonitis, arthritis, carpal tunnel syndrome). All subjects gave their written informed consent to participate in the study, which had been approved by the local scientific and ethics committee.

Experimental Setup

Google's open source MediaPipe hand landmark detection was used to record single-digit movement of the fourth (ring) finger in flexion and extension (Image 1). This was paired with musical notes from low pitch (joint in extension) to high pitch (joint in flexion). The remainder of the digits did not produce sound. Participants used a personal computer, or one provided by the investigational team, to interact with the web-based program. They were then asked to complete a Qualtrics survey to ascertain construct, face, content, and predictive criterion validity of the movement sonification program.

Image 1.

MediaPipe Hand Tracking.

Construct Validity

A recorded video of an investigator using the program was played and participants were required to describe in their own words how the program worked. Construct validity was established based on the ability of participants to identify that movements of a single digit was linked to musical parameters and distinguish between different finger positions (flexion and extension) based on produced sounds. Two reviewers (AP and MG) reviewed the statements individually to ascertain if they understood the assignment based on their descriptions, and a third reviewer was used to resolve disagreements.

Face Validity

Facilitated through the Qualtrics survey, the authors determined if participants found the program to reliably convert finger movement to sound, and if the sounds helped them understand different finger positions during movement.

Content Validity

Participants were asked if the program only produced sound to movement of the fourth (ring) digit. This allowed us to determine the program's ability to produce sound with single-digit movements that was not affected by other digits or video content.

Predictive Criterion Validity

After participants had explored the video tutorial, they were then shown an example of a sound recording (flexion vs extension) from the video and presented with options of multiple finger configurations that would produce that sound. This included distractor images and options for none or all the above. This was repeated eight times. Next, the participants were shown a muted video of a hand configuration, and they had to correctly predict the correct change in pitch that would be produced by the program. This was repeated eight times with distractor options of sounds not produced by the program. Predictive Criterion validity was assessed on the ability of participants to correctly predict the corresponding sounds to different finger configurations and vice versa.

Results

Participants (n = 19) had a mean age of 32.21 years (range 25-60). The sample included 10 male and 9 female participants. Recruitment was conducted by convenience sampling of healthy adults from the University of Calgary community.

Construct Validity

We assessed construct validity based on the ability of naïve participants to identify that movements of the fourth digit linked to the tone of the sound produced by the online program. Seventeen of 19 participants (89.47%) accurately identified that the sound produced by the online sonification program corresponded to flexion and extension of a single finger (the ring finger). Representative comments included: “Pitch goes up when the ring finger is flexed, pitch goes down when straightened,” and “Flexion of the fourth digit increased the note pitch while movement of other fingers had no effect.” Only two participants provided descriptions unrelated to the intended function.

After participants explored the program for themselves, they responded to the question “After exploring the program for yourself, how do you think this program works?” Seventeen of 19 participants (89.47%) accurately identified that the sound produced by the online sonification program corresponded to flexion and extension of a single finger. The remaining two participants did not answer correctly or provided a response that was not relevant to the question. Representative comments included “It is able to identify the fourth digit and respond with an increased frequency as the digit is flexed.” and “The pitch of the sound changes to be higher when you flex the fourth digit but not the other fingers.”

Face Validity

We assessed face validity by asking participants “Was there a relationship between sound and movement in the previous video” after they watched the demonstration video. This helped us understand if participants understood that the program to convert movement to sound, and whether then sounds help them understand the different finger positions during movement. All 19 participants (100%) responded “yes” to the question suggesting high face validity to the online movement sonification program.

Content Validity

Content validity was assessed with a series of questions administered after participants had watched the demonstration video and had a chance to explore the online program. First, we asked participants “Did the program pair sound to finger movements?” All 19 participants answered “yes” (100% correct; Figure 1). Next, we asked participants “What finger(s) produced sound when they moved? Select all that apply?” and provided them with the options to select digits 1-5. All participants selected the fourth digit (100% correct; Figure 1). Finally, we asked participants “Aside from finger movement, did anything else cause the program to produce sounds?” as a means of assessing that the content of our online program was specific to only the fourth digit. Similarly, all 19 participants selected “no” (100% correct; Figure 1).

Figure 1.

Content Validity.

Predictive Criterion Validity

We tested predictive criterion validity by presenting participants with a sound and asked them “Listen to the following sound and select from the options below.” We presented participants with several images of hand configurations and were asked to select the hand configuration that best corresponded to the sound. This was repeated eight times. Data was missing from one participant. Overall, the average percentage of correct responses across the eight trials was 95.14% (Range: 88.89-100.00%; Figure 2A).

Figure 2.

Predictive Criterion Validity. A) Participant Response Accuracy Across Eight Trials to Select Correct Hand Configuration Based on Audio. B) Participant Response Accuracy Across Eight Trials to Selected the Correct Sounds Based on Hand Configuration Seen in Video.

We tested predictive criterion validity a second way by presenting participants with a short demonstration video of a study investigator moving their finger (ie, flexion or extension) in the absence of sound. Participants were asked “Watch the following video and select the corresponding sound below.” Participants were presented with several options describing the pitch and any expected changes in pitch based on the movements observed in the video. This was repeated eight times. Overall, the average percentage of correct responses across the eight trials was 94.74% (Range: 84.21-100.00%; Figure 2B). Taken together, the results suggest that the online program has a high level of predictive criterion validity.

Discussion

This study validates the use of a novel movement sonification interface that uses Google's MediaPipe hand-tracking technology to map single-digit PIP joint motion to real-time auditory feedback. Our validation study was designed to assess construct validity, face validity, content validity, and predictive validity.^33,34 In essence, validity is the extent to which an instrument measures what it is meant to measure. Overall, our results demonstrate excellent construct, face, content, and predictive criterion validity. This supports the feasibility of using music sonification as an intuitive and accessible tool for augmented hand rehabilitation.

In our study, 89.47% of participants were able to explain in their own words that the pitch changes corresponded to flexion and extension of the fourth digit, confirming strong construct validity. In addition, 100% of participants in our study correctly identified that there was a relationship between sound and movement, confirming face validity of the program. This supports the intuitive design of the interface and coincides with other AR rehabilitation systems that demonstrate that humans naturally integrate auditory cues with motor actions when the sonification mapping is intuitive.^29,35

All participants in our study correctly identified that the fourth digit produced sound and that other digits did not trigger audio feedback, demonstrating excellent content validity. This addresses a major limitation of earlier sonification systems that used inertial sensor units that do not allow the capturing of finger or grasping movements.³⁶ In contrast, our study's findings are similar to those of a previous study by Dong et al, which demonstrates that MediaPipe's hand-tracking software enables precise digit and joint isolation.³²

In addition, participants in the current study correctly matched hand configurations with an average accuracy of 95.14% and correctly predicted sounds from muted videos with 94.74% accuracy. Both of which demonstrate strong predictive criterion validity. Predictive interpretation is central to motor learning as it enables users to anticipate correct movement patterns. Previous studies in neurologic rehabilitation show that predictive sonification improves movement timing, movement smoothness, and proprioception.^29,30,37

In the context of neurologic rehabilitation, particularly after stroke, prior work has shown that music-based sonification can meaningfully enhance upper-limb recovery. One study demonstrated improvements in manual dexterity, motor function, and decreased perceived pain during rehabilitation.²⁹ Similarly, Sholz et al, reported that music sonification in post-stroke patients undergoing upper extremity rehabilitation improved movement smoothness and reduced joint pain.³⁰ Of note, these studies focused on large-joint or multi-segment movements. Our study adds to the literature by demonstrating that single-digit PIP joint motion can be accurately captured and sonified, using a web-based interface. Participants intuitively understood the sound-movement mapping, highlighting the accessibility and usability of this software. Our work also expands on existing AR tools such as the DIGITS app, which demonstrates that finger range of motion can be reliably measured using a smartphone camera.³² Here, we demonstrated that using a no-hardware, webcam-based method, users can reliably perceive and predict sound mappings at the level of a single interphalangeal joint.

Despite the importance of hand rehabilitation for many common hand surgeries, including flexor and extensor tendon repair and hand fractures, adherence to rehabilitation protocols remains a major determinant to full functional recovery.^5,14,38,39 Several barriers to adherence are recognized, including limited access to therapy, lack of guidance during home exercises, and reduced motivation due to pain or monotony.^10,14 A prior study found that tailored information and recognition of gradual improvements play a large role in patient motivation and commitment to rehabilitation.¹⁴ In addition, AR and VR technologies have been demonstrated to enhance hand rehabilitation by providing real-time feedback, and gamifying therapy, which increases patient engagement and adherence.⁸ The current study adds to the evidence that our sonification tool may help address barriers to hand therapy adherence by (1) providing a browser-based platform that lowers geographic barriers to post-operative care, (2) providing real-time, joint-specific movement feedback and (3) increasing enjoyment, similar to gamified or AR-based rehabilitation tools.

Future work will evaluate the clinical efficacy of digit-level sonification in post-operative hand-injury populations. Randomized trials comparing sonification-based programs with standard writer or video-based protocols could assess functional outcomes, adherence, patient satisfaction, and barriers to care.

This study has several limitations. Firstly, it was conducted in healthy adults and excluded patients with learning disabilities or neurologic deficits. Therefore, results may differ in the general population. Our sample size of 19 individuals was small, and further studies will employ a larger number of individuals with less exclusions. Furthermore, survey-based studies are subject to individual compliance, and one of the participants failed to provide representative assessment in their description of the app affecting our construct validity. Finally, only a single digit in 2 configurations (flexion and extension) was used in this validation study. Other domains such as multi-digit movements, velocity, or joint angles were not assessed in this study as would be used in hand rehabilitation. Future studies would aim to target these areas.

Conclusion

This study validates the use of a novel movement sonification interface that uses Google's MediaPipe hand-tracking technology to map single-digit PIP joint motion to real-time auditory feedback. Overall, our results demonstrate excellent construct, face, content, and predictive criterion validity. This supports the feasibility of using music sonification as an intuitive and accessible tool for augmented hand rehabilitation.

Footnotes

Contributory Roles

Supervision: JY, MR, AK. Conceptualization: AP, MJG, RTM, RH, MR, JY. Investigation: AP, MJG, RTM. Methodology: AP, MJG, RTM, RH, MR, JY. Validation: AP, MJG, RTM, RH, MR, JY. Formal Analysis: AP, MJG, RTM. Writing—Original Draft: AP, MJG, EY. Writing—review & editing: AP, MJG, RTM, EY, RH, MR, AK, JY.

Financial Disclosure Statement

Authors do not have an affiliation (financial or otherwise) with a pharmaceutical, medical device, or communications organization. A research grant was provided by the Department of Surgery, Division of Plastic Surgery, University of Calgary, Calgary, AB, Canada

Participant Consent Statement

Informed, written consent was obtained from all participants included in the study regarding recruitment, participation, and the publication of their information.

Data Availability

Participant data and all findings of this study are available from the corresponding author upon reasonable request. All data are de-identified and stored in accordance with institutional ethics requirements.

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Ethics Statement

Ethical Approval was provided by the Conjoint Health Research Ethics Board (CHREB), University of Calgary, Alberta, Canada. REB25-0645.

Funding

The authors received no financial support for the research, authorship, and/or publication of this article.

ORCID iDs

Alexander Platt

Martin J. Giuffre

Emma Yanko

Maleka Ramji

References

Thien

Becker

Theis

. Rehabilitation after surgery for flexor tendon injuries in the hand. Cochrane Database Syst Rev. 2004(4):CD003979. doi:10.1002/14651858.CD003979.pub2

Guzelkucuk

Duman

Taskaynatan

Dincer

. Comparison of therapeutic activities with therapeutic exercises in the rehabilitation of young adult patients with hand injuries. J Hand Surg Am. 2007;32(9):1429-1435. doi:10.1016/J.JHSA.2007.08.008

Angermann

Lohmann

. Injuries to the hand and wrist: A study of 50,272 injuries. J Hand Surg Br. 1993;18(5):642-644. doi:10.1016/0266-7681(93)90024-A

Trybus

Lorkowski

Brongel

Hładki

. Causes and consequences of hand injuries. Am J Surg. 2006;192(1):52-57. doi:10.1016/J.AMJSURG.2005.10.055

Peters

Jha

Ross

. Rehabilitation following surgery for flexor tendon injuries of the hand. Cochrane Database Syst Rev. 2021(1):CD012479. doi:10.1002/14651858.CD012479.pub2

Goggins

Williams

. Rehabilitation of the hand. Orthop Trauma. 2019;33(1):62-65. doi:10.1016/J.MPORTH.2018.11.008

Blanquero

Cortés-Vega

Rodríguez-Sánchez-Laulhé

Corrales-Serra

Gómez-Patricio

Díaz-Matas

. Feedback-guided exercises using tablet touchscreen vs paper exercise programs. J Physiother. 2020;66(4):236-242. doi:10.1016/j.jphys.2020.09.012

Pereira

Prahm

Kolbenschlag

Oliveira

Rodrigues

. Application of AR and VR in hand rehabilitation: A systematic review. J Biomed Inform. 2020;111:103584. doi:10.1016/J.JBI.2020.103584

Reitan

Dahlin

Rosberg

. Patient-reported quality of life following traumatic hand injury. Health Qual Life Outcomes. 2019;17(1):148. doi:10.1186/s12955-019-1215-9

10.

Krishnan

Chung

. Access to hand therapy following surgery in the United States. Hand Clin. Published online 2020;36(2):205-213. doi:10.1016/J.HCL.2020.01.006

11.

Svingen

Rosengren

Turesson

Arner

. A smartphone application to facilitate adherence after flexor tendon repair. Clin Rehabil. 2021;35(2):266-275. doi:10.1177/0269215520962287

12.

Then

Shivdas

Tunku Ahmad Yahaya

Ab Razak

Choo

. Gamification in rehabilitation of metacarpal fracture. J Hand Ther. 2020;33(2):235-242. doi:10.1016/J.JHT.2020.03.029

13.

Bittel

Williams

Elazzazi

. Improving exercise performance with an accelerometer-based smartphone app. Am J Phys Med Rehabil. Published online 2017;96(5):307-314. doi:10.1097/PHM.0000000000000618

14.

Svingen

Arner

Turesson

. Patients’ experiences of flexor tendon rehabilitation in relation to adherence. Disabil Rehabil. 2023;45(7):1115-1123. doi:10.1080/09638288.2022.2051081

15.

Gialanella

Ettori

Faustini

Baratti

Bernocchi

Comini

. Home-based telemedicine in chronic neck pain. Am J Phys Med Rehabil. 2017;96(5):327-332. doi:10.1097/PHM.0000000000000610

16.

Cranen

Groothuis-Oudshoorn

CGM

Vollenbroek-Hutten

MMR

IJzerman

. Preferences for web-based telerehabilitation. J Med Internet Res. 2017;19(1):e26. doi:10.2196/jmir.5951

17.

Yedlinsky

Peebles

. Telemedicine management of musculoskeletal issues. Am Fam Phys. 2021;103(3):147-154.

18.

Phuphanich

Sinha

Truong

Pham

. Telemedicine for musculoskeletal rehabilitation. Phys Med Rehabil Clin N Am. 2021;32(2):319-353. doi:10.1016/J.PMR.2020.12.004

19.

Zhou

Wang

Luan

. Exercise rehabilitation in breast cancer survivors. Support Care Cancer. Published online. 2023;31(5):307. doi:10.1007/s00520-023-07765-9

20.

Melillo

Chirico

De Pietro

. Virtual reality rehabilitation systems for cancer survivors. Cancers (Basel). Published online 2022;14(13):3163. doi:10.3390/cancers14133163

21.

Chen

Zhou

Luan

Peng

. Virtual reality-based interventions for upper limb rehabilitation in breast cancer patients. Games Health J. 2024;13(3):149-163. doi:10.1089/G4H.2023.0040

22.

Laver

Adey-Wakeling

Crotty

Lannin

George

Sherrington

. Telerehabilitation services for stroke. Cochrane Database Syst Rev. 2020(1):CD010255. doi:10.1002/14651858.CD010255.pub3

23.

Johansson

Wild

. Telerehabilitation in stroke care–a systematic review. J Telemed Telecare. 2011;17(1):1-6. doi:10.1258/jtt.2010.100105

24.

Burke

Mcneill

Charles

Morrow

Crosbie

Mcdonough

. Augmented reality games for upper-limb stroke rehabilitation. VS-Games Conference. 2010:75-78. doi:10.1109/VS-GAMES.2010.21

25.

Massetti

da Silva

Crocetta

Guarnieri

de Freitas

Bianchi Lopes

. Clinical utility of virtual reality in neurorehabilitation. J Cent Nerv Syst Dis. 2018;10:1179573518813541. doi:10.1177/1179573518813541

26.

Dunn

Yeo

Moghaddampour

Chau

Humbert

. Virtual and augmented reality in phantom limb pain. NeuroRehabilitation. Published online 2017;40(4):595-601. doi:10.3233/NRE-171447

27.

Effenberg

Melzer

Weber

Zinke

. MotionLab sonify. Information Visualisation Conference. 2005:17-23. doi:10.1109/IV.2005.84

28.

Nown

Upadhyay

Kerr

Andonovic

Tachtatzis

Grealy

. Real-time movement sonification systems. IEEE Rev Biomed Eng. 2023;16:672-686. doi:10.1109/RBME.2022.3187840

29.

Raglio

Panigazzi

Colombo

Tramontano

Iosa

Mastrogiacomo

. Hand rehabilitation with sonification. Sci Rep. 2021;11(1):7237. doi:10.1038/s41598-021-86627-y

30.

Scholz

Pirzer

Schneider

Rollnik

Großbach

. Sonification as a stroke rehabilitation strategy. Front Neurosci. 2014;8:332. doi:10.3389/fnins.2014.00332

31.

Starkov

Jochems

Rijsdijk

Snellenberg

Stoffels

Zaidi

. Sonifying motor skills with Pizzicato. CIG Conference. 2024:1-8. doi:10.1109/COG60054.2024.10645594

32.

Dong

Shin

Banerjee

Masschelein

Davidson

. DIGITS app-smartphone augmented reality. Comput Methods Biomech Biomed Eng Imaging Vis. 2021;10(4):375-382. doi:10.1080/21681163.2021.1998927

33.

APA Dictionary of Psychology. APA; 2025.

34.

Bull

Teede

Watson

Callander

. Selecting and implementing patient-reported outcome and experience measures. JAMA Health Forum. 2022;3(4):e220326. doi:10.1001/jamahealthforum.2022.0326

35.

Fabiani

Bresin

Dubus

. Interactive sonification of expressive hand gestures. J Multimodal User Interfaces. 2012;6(1):49-57. doi:10.1007/s12193-011-0076-2

36.

Schmitz

Bergmann

Effenberg

Krewer

Hwang

Müller

. Movement sonification in stroke rehabilitation. Front Neurol. 2018;9:389. doi:10.3389/fneur.2018.00389

37.

Schaffert

Janzen

Mattes

Thaut

. Sound and movement in sports and rehabilitation. Front Psychol. 2019;10:244. doi:10.3389/fpsyg.2019.00244

38.

Jones

Jupiter

Lalonde

. Common fractures and dislocations of the hand. Plast Reconstr Surg. 2012;130(5):722e-736e. doi:10.1097/PRS.0b013e318267d67a

39.

Sameem

Wood

Ignacy

Thoma

Strumas

. Rehabilitation protocols after extensor tendon repair. J Hand Ther. 2011;24(4):365-373. doi:10.1016/j.jht.2011.06.005

Validation Study on Movement Sonification of the Hand as a Rehabilitation Tool

Abstract

Keywords

Introduction

Methods

Subjects

Experimental Setup

Construct Validity

Face Validity

Content Validity

Predictive Criterion Validity

Results

Construct Validity

Face Validity

Content Validity

Predictive Criterion Validity

Discussion

Conclusion

Footnotes

Contributory Roles

Financial Disclosure Statement

Participant Consent Statement

Data Availability

Declaration of Conflicting Interests

Ethics Statement

Funding

ORCID iDs

References