Abstract
Augmented Reality (AR) is increasingly used in various remote rehabilitation applications for patients with functional limitations. Providing multimodal feedback using auditory, visual, or haptic modalities can offer effective guidance to the patient while they perform self-supervised rehabilitation exercises. This review examines the current uses of AR with multimodal feedback in upper extremity rehabilitation applications built for stroke and Parkinson’s disease (PD) patients by synthesizing nine peer-reviewed studies. The review identified the improvement and engagement that AR-based rehabilitation provides to such patients. The current body of literature shows that AR-based rehabilitation with multimodal feedback is promising. However, technology and its application for rehabilitation use are still in their infancy. More research is required in selecting effective feedback modalities and clinically-relevant health outcome measures and in providing diverse rehabilitation tasks designed specifically for stroke survivors and PD population.
Introduction
Traditional rehabilitation programs (e.g., in-person visits to physical therapists) for stroke and Parkinson’s disease (PD) patients have several limitations that can impact the accessibility and effectiveness of the programs. First, these programs can be costly, making it difficult for individuals with limited financial resources to participate regularly (Sevcenko & Lindgren, 2022). Second, these in-person programs can be time-consuming and inconvenient for individuals who live far from therapy centers or have limited mobility and transportation options, making the programs less accessible (Dorsey et al., 2016; Phan et al., 2022). This can also limit the number of therapy sessions an individual can attend, making it challenging for them to make frequent visits, which are important to make good progress. Third, traditional rehabilitation programs often lack engagement and can be repetitive and boring, making it difficult for individuals to stay motivated and involved, thus adversely affecting their adherence to the programs (Burdea, 2003). These limitations make it challenging for individuals with stroke or PD to access and benefit from traditional in-person rehabilitation programs.
The upper extremities are essential for performing daily activities, and impairment in these limbs can significantly impact a person’s quality of life (Magermans et al., 2005). Stroke and PD conditions often result in motor impairment in the upper extremities, such as weakness, spasticity, and reduced coordination (O’Dwyer et al., 1996; Williams et al., 2013). So, upper extremity rehabilitation is often a focus of rehabilitation for stroke and PD patients as this can significantly improve their ability to perform daily activities, enhance their independence, and improve their quality of life (Shin et al., 2016).
AR has the potential to make rehabilitation programs more accessible, engaging, and effective by providing a stimulating and enjoyable experience (Monge & Postolache, 2018). Such experience can enhance motivation and adherence to rehabilitation programs (Burke et al., 2010). Additionally, AR can be used to create virtual scenarios that mimic real-life scenarios (i.e., grating a carrot, lifting and moving cups); thus, providing opportunities to practice the skills in virtual environments and apply them in the real world (Alamri et al., 2009; Colomer et al., 2016). This can help to increase the transfer of skills to real-life situations, improving the patient’s daily functioning and quality of life. Furthermore, rehabilitation programs with AR can be delivered in a remote or home-based setting, reducing the need for in-person therapy sessions and making it easier for individuals to access care (Lee et al., 2022; Munroe et al., 2016).
Additionally, real-time multimodal feedback (e.g., auditory, visual, or haptic) can be used in AR-based rehabilitation programs to provide effective feedback on performance. Multimodal feedback has been known to improve motor learning and skill acquisition (Islam et al., 2022; Islam & Lim, 2022; Sigrist et al., 2013), which can ensure the generalization and retention of motor skills (Gorman & Gustafsson, 2022). By incorporating multimodal feedback with AR, patients can receive real-time, objective assessments and guidance on their performance, which can improve rehabilitation outcomes (Vieira et al., 2015).
This study presented herein aimed to examine the current state of novel AR technology and multimodal feedback for stroke and PD rehabilitation in upper extremity rehabilitation. Specifically, we aim to understand: (1) the current state of the science of using AR systems with multimodal feedback for aiding the upper extremity functions and the effectiveness of such systems in rehabilitation outcomes; and (2) research gaps and opportunities in current technology.
Methods
Database and Search Strategy
Five databases were considered for this review: Pubmed, Proquest, Web of Science, Science Direct, and IEEE Xplore. Articles published from January 2000 to December 2022 were considered for this study. The following combination of keywords were used: [“Augmented Reality” OR “AR”] AND [“Multimodal” OR “Cross-modal” OR “Real-time” OR “Intervention”] AND [“Rehabilitation” OR “Therapy” OR “Treatment” OR “Training”] AND [“Parkinson’s disease” OR “Stroke”]. Full-length peer-reviewed journal articles and conference proceedings published in English and containing the keywords were selected initially for the title and abstract review.
Inclusion/Exclusion Criteria
After removing the duplicates (n = 207), initially selected papers (n = 1,633) were reviewed in two stages. In the first stage, the title and abstract of the papers were reviewed for their scope. The exclusion criteria in this stage were any study that did not have an AR environment, studies that did not perform physical rehabilitation or similar activities, and studies that did not have stroke or PD patients as their participants or target population. Studies with healthy participants were included if the rehabilitation program was intended for stroke or PD patients. From this first stage review, 243 studies were selected for full-text review. In the second stage, a full-text review was performed to filter papers that did not meet the criteria of this scoping review. Exclusion criteria include papers with no (real-time) multimodal feedback and rehabilitation not aimed at the upper extremity. After the second stage review, seven papers were selected for the final scoping review. References of the finally selected papers for the scoping review were also tracked, which provided an additional two papers, totaling nine papers for this review.
Data Synthesis of the Selected Studies
Selected studies were thoroughly reviewed, and the following information was extracted: general information including author(s), year of publication, title, journal, study participant characteristics and sample size, description of the rehabilitation task, targeted body parts, AR experience that participants received (i.e., what objects were superimposed, how the environment was projected) and key outcomes of the study.
Results
Among the nine reviewed studies, seven (78%) were rehabilitation programs for stroke patients, and two (22%) were for both stroke and PD patients (see Table 1). Three of the studies (33%) used healthy participants for their study, but the target population was stroke patients (Alamri et al., 2009; Aung & Al-Jumaily, 2012; Khademi et al., 2013). One study used healthy participants as a control group and compared it with both stroke and PD patients’ performance (Bank et al., 2018).
Summary of the Studies Reviewed in this Paper (n = 9).
A = auditory; V = visual; H = haptic.
Three studies used long-term intervention programs lasting one to six weeks (Boos et al., 2011; Colomer et al., 2016; Hoda et al., 2015). The other six studies were single-session studies. Two of the studies had a physical therapist present to supervise the task activities (De Assis et al., 2016; Colomer et al., 2016). Two of the studies measured the baseline performance of the participants (pre-test) and compared the results after the intervention (post-test) (De Assis et al., 2016; Hoda et al., 2015).
The AR systems used in the reviewed studies superimposed virtual objects (i.e., mug, box) that needed to be manipulated, the path that needed to be followed, and the impaired/unimpaired body parts. For example, in the study by Alamri et al. (2009), the AR rendering component receives the real-world scene captured by a camera. It computes the pose (position and orientation) of a special marker in the scene. This pose is used to superimpose virtual objects onto the real scene. The subject is asked to move a real kitchen object (a mug) back and forth numerous times to a shelf along a guided path presented in front of them. In the study by Boos et al. (2011), participants viewed only the unimpaired hand and its mirror image, superimposed on the impaired hand’s position. Both virtual hands were controlled by the movement of the unimpaired hand, showing the participants an image of their impaired hand moving normally. Two studies used auditory, visual, and haptic feedback (Boos et al., 2011; Colomer et al., 2016), whereas the other seven used auditory and visual feedback.
Different outcome measures were used for the reviewed studies to determine the effectiveness of the rehabilitation program. Seven of the studies used objective measures: (1) kinematic or kinetic measures of the upper extremities, including force and peak velocity of the impaired body parts, joint angle variability, and reach speed; (2) completion time; (3) AR game scores; and (4) clinically-relevant measures, such as Fugl-Meyer scale (Gladstone et al., 2002), Action Research Arm (ARM) Test (Yozbatiran et al., 2008), Wolf motor function test (Wolf et al., 2001), box and blocks test (Mathiowetz, Volland, et al., 1985), and nine hole peg test (Mathiowetz, Weber, et al., 1985).
Three of the studies used subjective measures, that is, game experience questionnaires and System Usability Scale (SUS) (Grier et al., 2013), to determine the effectiveness and usability of the rehabilitation program. Among the studies that objectively measured the performance, four reported significant improvement in their functions after receiving the rehabilitation training (De Assis et al., 2016; Colomer et al., 2016; Hoda et al., 2015; Khademi et al., 2013).
Discussion
The aim of this scoping review was to synthesize how AR technology is currently being used in rehabilitation applications designed for stroke and PD patients’ upper extremity rehabilitation with multimodal feedback. This review found that the current use of AR-based rehabilitation systems is limited to small, one-site studies, wherein some of which are pilot studies or use healthy participants. This demonstrates that AR-based rehabilitation technology is still in the early stages of development. Overall the systems that were presented in this review identified either improvement with the AR-based rehabilitation or determined the system was safe to use by the target population, which shows promising future implications for AR-based rehabilitation systems.
The review showed that most of the current AR systems are designed to engage users in repetitive activities or movements to enhance specific functional aspects. There is potential for AR technology to be used beyond repetitive activities or movement, such as to create simulations of real-world environments, allowing patients to practice daily activities (i.e., cooking and getting dressed) (Khademi et al., 2013) and engaged to socially interactive home-based rehabilitation sessions to reduce the feeling of isolation among patients.
Most of the studies (7 out of 9) used only auditory and visual feedback. However, multimodal feedback systems beyond auditory and visual (i.e., vibrotactile, olfactory, and force feedback) improve motor learning in virtual environments (Islam & Lim, 2022). Proper use of multimodal feedback in AR-based rehabilitation systems can potentially improve the program’s effectiveness, which needs to be considered in future studies.
Through this review, we identified a developing body of evidence in evaluating the use of various forms of AR technology for stroke and PD rehabilitation. Although the scope of this review did not extend to the evaluation of functional outcomes, it identified user motivation and engagement in rehabilitation improved with AR. Future studies should be more focused directly on the target population as we have observed gaps in performance improvement between healthy and target populations in some of the reviewed papers.
Footnotes
Declaration of Conflicting Interests
The author declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author received no financial support for the research, authorship, and/or publication of this article.
