Abstract
Movement sonification (i.e., translating kinematic measures into sound, providing auditory feedback), can enhance motor performance and (re)learning across both clinical and sports contexts. However, existing sonification systems vary widely, potentially affecting efficacy. Here, we categorize sonification systems based on the utilized sound (type) and onset (trigger) to elucidate potential differences in movement-, intuitiveness-, and motivation-related outcomes. Movement sonification systems, as described in 101 included studies, were classified into six sound types (specifically parameter mapping, musically informed, single tone, rhythmic, music-based, and environmental) and four trigger types (event-, error-, trajectory-based, and continuous), and effects on outcomes were compared. Overall, positive effects on all outcomes were reported for all system types and triggers, with no clear pattern of advantages for any, except that learning retention is more consistently reported for triggers involving explicit learning strategies. In general, more subtle movement learning may benefit from more direct movement-sound coupling, implicit learning, and multi-component mappings. Additionally, systems providing rhythmic sounds (musically informed, single tone, rhythm, music-based) were more commonly used for periodic movements. Intuitiveness is rated highest for parameter mapping and rhythmic sonification, and motivation strongest for musically informed sonification, although all systems increase autonomous motivation, compared to no sonification. In conclusion, movement sonification systems generally improve performance but arguably work differently for different movements. Accordingly, musically informed sonification may be preferable for rehabilitation settings where motivation is crucial, while parameter mapping sonification that inform users about subtle kinematic aspects may especially support learning highly skilled movements. Standardizing terminology in future work will facilitate systematic cross-system comparisons.
Introduction
In the present review, movement sonification refers to the process of providing real-time auditory feedback of executed movements, by translating kinematic as well as dynamic movement parameters into sounds (e.g., Hermann et al., 2011). Crucially, systems that sonify movement allow for the manipulation of the amount and type of auditory feedback of movement, thus providing a context, which is highly influential to the formation of movement representations (Ruitenberg et al., 2012, 2015). Prior work has reported promising results regarding the use of sonification to assist motor performance or (re)learning in clinical populations (e.g., stroke, Parkinson’s disease) as well as athletes (for a review, see Schaffert, Janzen, et al., 2019). Movement sonification leads to strong auditory-motor associations, even outperforming the reported benefits of auditory cueing techniques, which involve moving to sound rather than creating sound with movement. Additionally, movement sonification has potential in aiding physical rehabilitation, based on positive results regarding movement quality and control, increased movement and body-awareness, as well as improved performance (for a review, see Guerra et al., 2020). However, we argue that the requirements of sonification systems may differ significantly according to an athlete’s or patient’s individual needs, which may vary from receiving precise feedback on performed movements to encouraging movement more generally. The present systematic review aims to identify potentially relevant requirement differences by categorizing sonification systems based on how the feedback is created, and summarize the corresponding outcomes related to movement quality and learning, as well as intuitiveness of the system, and motivational value.
Within research on augmented feedback for learning and improving motor skills, two main approaches are described, namely augmented feedback and modeling. The former represents adding additional feedback to the movement providing performance-related information, whereas in the latter, an example model is provided to be imitated by the performer (for a review see Sors et al., 2015). Movement sonification can be effectively applied in either of these approaches, sometimes also using concurrent visual stimuli to create more holistic augmented feedback (Effenberg et al., 2016; Ghambari et al., 2024; Parimi et al., 2024; Pizzera et al., 2017; Ramezanzade, 2020; Ramezanzade et al., 2017). However, the auditory modality is suggested to be more practical as it hinders the processing of sensory afferences to a lesser extent, as compared to visual feedback (Sigrist et al., 2013). As the aim of the present review is to evaluate unisensory auditory movement sonification outcomes based on augmented feedback, we use the term sonification accordingly.
To date, literature that systematically categorizes sonification systems is scarce. The most salient aspect of sonification systems, namely what kind of sound is used (referred to here as the sonification type), has not received much research attention. One can argue that the more fine-grained or specific the auditory feedback is, the easier it is to understand and use to improve movement. For instance, directly mapping a kinematic parameter to a sound parameter can magnify subtle aspects of movement that are difficult to perceive with proprioception, however, these sounds may not be very pleasant to listen to and this may thus impact the motivation to use a system. For the purpose of the current review, we aim to categorize the utilized sounds, thus distinguishing different types of sonification systems. Prior work has described system distinctions that identify several of these types. Specifically, when directly mapping a movement feature to an auditory feature as described above, which we here term parameter mapping sonification (PMs), movement execution constantly adjusts continuous sound parameters (e.g., pitch, brightness, loudness) of pure sine waves or pink/white noise, thus directly translating movement to sound. On the other hand, Schaffert, Janzen, et al. (2019) describe a relatively new sonification type, here referred to as musically informed sonification (MIs), where sounds are informed by musical information, pre-designed to make more musically coherent or pleasant sounds. For example, instead of using continuous pitch as in PMs, discrete units of pitch are used as a scale, resulting in a melody played through movement execution. This may make sounds easier to perceive and more pleasant to listen to (Brown et al., 2003), yet necessarily reducing the directness or resolution of the movement-sound coupling in comparison to direct parameter mapping, potentially negatively influencing the ease to understand, learn, and use a sonification system, which we here refer to as intuitiveness. However, as MIs systems may sound more pleasant, they may also increase the user’s motivation.
The extent to which auditory feedback is perceived as motivational has been argued to require clarification in the context of learning with sonification systems (Bevilacqua et al., 2016), which may be affected by system type. Motivation is often defined based on the self-determination theory (Ryan & Deci, 2000), including several sub-categories of motivation. Specifically relevant for the current question, autonomous or intrinsic motivation refers to behavioral drive based on inherent satisfaction and pleasure, the individual’s sense of self (potentially through expressiveness), or when it is personally important to the individual, possibly through a learning objective (Ryan & Deci, 2000). This kind of motivation has been associated with long term maintenance of exercise adherence (Teixeira et al., 2012). Moreover, increased positive mood, exercise enjoyment, and feelings of power may also improve movement performance (Biagini et al., 2012; Hsu et al., 2015; Stork et al., 2015). As such, the potential affective element of sonification types using musical sounds might support both quantity and quality of movement. During exercise, music listening can reduce perceptions of fatigue and exertion due to dissociation and distraction (Ballmann et al., 2019; Bood et al., 2013; Boutcher & Trenske, 1990), and can increase motivation and effort resulting in improved performance (Ballmann et al., 2019, 2021; Karow et al., 2020). A recent systematic review on music-based interventions supports the notion that motivation serves as a fundamental mechanism in their efficacy (Dimitriadis et al., 2023). Taken together, music listening may increase both the motivation to move, as well as movement performance. However, this does not include the agency involved in creating sounds yourself. As an example of a sonification system that uses fully formed music that rewards any movement, Fritz et al. (2013) show that when exercising, creating musical sounds results in higher reported positive mood in healthy participants compared to passive music listening while exercising. In this case, no feedback information about the movement quality is provided, but the sonification is used to enhance endurance. This implies that sound is used differentially when aimed at motivational value versus enhancing movement precision.
Another perspective on movement sonification focuses on what initiates the sound, here referred to as the sonification trigger, and two main approaches have previously been distinguished for learning (Bevilaqua et al., 2016). Firstly, an explicit learning approach informs the user in real-time about errors in the executed movement in comparison to the predefined movement as prescribed by the system (i.e., error-based triggers). Secondly, an implicit learning approach involves the continuous sonification of movements, without or irrespective of a target movement (i.e., continuous trigger), and the mapping between movement and sound is acquired through exposure. In these approaches, sonification triggers are the mechanism through which the sonification system initiates or adjusts the sound, which subsequently serves as feedback to the user. We expect that additional triggers will emerge from the existing literature. Additionally, it is plausible that a sonification system incorporates both continuous and event-based triggers, leading to parallel implicit and explicit learning strategies. A previous study comparing explicit and implicit learning strategies found that only explicit feedback leads to 2-week retention of increased complex motor skill through improved decision-making in novices (Lola et al., 2021). We therefore propose that variations in sonification triggers may have differential outcomes regarding the performance or effectiveness of the system.
To make evidence-based practical and clinical recommendations, factors that are important for specific user groups should be considered. For instance, athletes are likely already intrinsically motivated to move, but would like to improve specific movement aspects, in order to improve performance. As adjusting already learnt movements may require small, precise changes to improve performance, system requirements may relate closely to the needed directness of movement-sound coupling (intuitiveness and informativeness). Conversely, system requirements for rehabilitation settings may be mainly related to motivational aspects (e.g., Rapolienė et al., 2018). Therefore, systems for rehabilitation may have to focus more on increasing motivation, potentially through the use of musical sounds.
The overarching goal of the present review is to provide a structured overview of the different movement sonification systems as described in the literature. In this way we seek to provide clear practical and clinical recommendations on which sonification type to use for which setting, including clear terminology which may create more homogeneity in future research. The first aim is to categorize sonification systems by type (sound) and trigger (onset). The second aim is to evaluate the findings for specific systems in terms of movement outcomes, including learning retention, as well as subjective ratings of intuitiveness, and motivation for these different sonification systems, informing possible recommendations. While not the main focus of the current review, it is expected that sonification in general shows positive effects on movement-related outcomes. Regarding the sonification types and movement outcomes, we expect that direct movement-sound coupling techniques, such as in PMs, are most efficient when a user’s goal is to improve movement quality or technique. We further expect that explicit learning-related sonification triggers will show better immediate and longer-term retention than implicit strategies. Regarding user experience, it is expected that participants will rate PMs as more intuitive (or understandable), also due to a more direct coupling, while MIs and MBs will lead to more positive motivation- or preference-related results, as they provide more musical sounds. Finally, as the importance of personalization in clinical recommendations is gaining interest, we will also explore the reported effects of any individual differences mentioned in the literature, to the extent that they are included.
Methods
Search Strategy
The literature review was conducted based on the PRISMA protocol, the checklist is provided as Supplemental Material S1. A comprehensive literature search without time restrictions was performed in 10 databases (PubMed, Web of Science, Embase, Cochrane Library, Emcare, PsycINFO, Academic Search Premier, IEEEXplore, ACM Digital Library and Google Scholar) on 19 April 2023. The search terms were based on keywords deemed to be relevant; ‘movement sonification’ (i.e., motor, audio, auditory, auditory-motor, augmented, concurrent, interactive, online, real-time, sound) OR ‘sonification AND music’ (e.g. music therapy, music supported, audio feedback, biofeedback, concurrent feedback, online acoustic information) OR ‘movement (e.g. motor, gait, locomotion, motor performance, motor skill, motor learning, sports) AND audio feedback’ (e.g. auditory, acoustic, augmented, auditory displays, feedback system, biomechanical biofeedback, external biofeedback). The complete search terms for each search engine are provided as supplemental materials (S2).
Initially, 1662 studies were identified; after removing duplicates, 905 abstracts were retained. Based on title and abstract screening, 711 records were excluded for not meeting the inclusion criteria (described below), leaving 194 full-text articles. Following a detailed eligibility assessment, 93 articles were excluded as they did not align with the objectives of the review. Ultimately, 101 studies were included. The flow chart of article inclusion is presented in Figure 1. Flow diagram illustrating the article selection for the systematic review
Selection of Studies
We included only experimental studies on auditory movement sonification, reporting measurable movement or motivation-related outcomes in adult human subjects from both healthy and clinical populations. Reviews, meta-analyses, dissertations, and studies describing system prototypes without reporting empirical data on motor or non-motor outcomes were excluded.
The abstract screening phase was executed by two independent investigators (M.M.P. & A.P.) to control for selection bias and to mitigate human error. Disagreements were resolved by including a third investigator (M.C.).
At the full text screening phase, which was executed by three investigators (M.C., M.M.P & A.P.), an additional criterion was added, namely to focus on movements potentially included in sports and rehabilitation, excluding studies on writing or surgery. Additionally, we included only sonification systems designed specifically to enhance movement aspects, rather than those created to enhance creativity in movement, as well as in sound, mainly excluding studies regarding dancing or playing a musical instrument. These decisions were made in order to reduce the number of studies, as well as to facilitate more effective comparisons between studies.
In the data extraction phase, evidence from the included articles was coded by three investigators (M.C., M.M.P., & A.P.). Additional emerging issues related to inclusion were resolved in team meetings.
Classification of Sonification Systems
Identified Sonification System Types and Triggers
Data Extraction and Synthesis
The extraction of data from the included papers comprised author name, participant population, sonification type and trigger as described above, as well as their movement-, intuitiveness-, and motivation-related outcomes. To address our main aim of evaluating the plausible relationship between the type of sonification and their movement-, intuitiveness- and motivation-related results, outcomes were coded according to these three categories for each study. To address the secondary aim, findings on immediate and delayed retention were extracted where available, in order to evaluate their possible relationship with the sonification trigger, using im- or explicit learning strategies. Finally, to explore potential influences of user diversity on sonification system use or efficacy, any correlates of individual differences were registered.
Results
Sonification Systems: Types and Triggers
Overview of the Number of Observed Type-Trigger Associations
Note. % are rounded values. Abbreviations: Parameter mapping (PMs), Musically informed (MIs), Single tone (STs), Rhythmic (Rs), Music-based (MBs), and Environmental sonification (Es). Given that 23 papers discussed multiple systems, the total number of systems exceeds the total number of included papers.
Motor and Non-Motor Outcomes per Sonification Type
In the following, we discuss sonification systems by type, reporting motor and non-motor outcomes while grouping over different triggers. These triggers, referring to the mechanism of sound initiation in sonification systems, were initially identified as two categories (Bevilacqua et al., 2016), namely error-based triggers, informing the user about errors or deviations of executed movement from a predefined movement reference, and continuous triggers, informing the user continuously, without any reference. These two categories mainly differ in the use of explicit and implicit learning strategies, respectively. Based on the included studies, we here describe two additional trigger categories. One concerns event-based triggers, which are comparable to error-based triggers, but instead of indicating an error inform the user about successful movement completion based on a predefined movement. Another concerns a combination of explicit and implicit learning strategies, continuously sonifying movement, but also refer to a predefined movement. For example, Nikmaram et al. (2019) evaluated a continuous trigger whereby six distinct notes of a C major scale were played by vertical movements (y-axis), brightness (different instruments) was adjusted by moving horizontally (x-axis), and volume was adjusted by moving on the z-axis. In addition, they asked participants to play a specific melody. Thus, participants were informed both implicitly, due to continuously sonified movements as described above, as well as explicitly, due to both error- and event-based triggers representing mistakes and success in playing the specific melody, further referred to as trajectory triggers. In the description of the following sonification system types, triggers are specified for all systems in the overview tables.
Parameter Mapping Sonification (PMs)
Overview of Motor and Non-motor Findings for Parameter Mapping Sonification (PMs) Systems, Grouped by Trigger
Note. HP = Healthy participants, AT = Athletes, SP = Stroke patients, Amp = Amputees, Exp = experiments, ➢ = motor outcomes, ○ = subjective outcomes, ■ = retention outcomes, ↑ = significant increase, ↑* = trend toward significant increase, ↓ = significant decrease, ✓ = retention, Ø = no significant difference/retention.
PMs: Movement Outcomes
As presented in Table 2, 28/128 (22%) of the included papers are categorized as PMs systems, of which 16/28 (57%) are compared to a control condition without sonification, and the remaining 12 systems are compared to other systems, and are described below (Table 9). Table 3 lists the motor outcomes of PMs systems compared to control, of which 14 (88%) show improvements, none show only degradations or null results, and 2 (12%) are mixed. Aspects of performance increases were found in all studies; including increased movement (force and/or velocity) and reduced errors for fine motor (hand) movements (Boyer et al., 2017, 2020; Guo et al., 2022) but not force variability (Guo et al., 2022). For gross motor movement, such as in sports, performance increases include less variability in timing of golf swing movements (O’Brien et al., 2020a), improvement of specific technique in specific speed-skating technique (Stienstra et al., 2011) and bicep curls (Yang & Hunt, 2015) as compared to a control condition without feedback, but also reduced performance of rowing, comparing concurrent to terminal feedback (Sigrist et al., 2013). For other movements such as gait or isolated limb movements, specific kinematic improvements were also found in knee repositioning (Ghai et al., 2018), walking cadence (Pang & Feltham, 2022), and trunk swaying (Giansanti et al., 2009). For balance, stance was either improved (Wannsted & Herman, 1978), only improved as a trend (Tillman et al., 2020) or not changed (Mullineaux et al., 2012). For stroke survivors’ arm movements, improved movement quality (Robertson et al., 2009) and decreased errors (Fujii et al., 2016) are reported. Overall, various positive outcomes of PMs were reported for a large range of movements including both fine and gross movements in both healthy and patient populations, with the least convincing evidence for stance balance, although the goals in these studies were different.
In terms of learning retention, only three studies included these measures. After removing the sonification, results showed 15 min and 24h retention of proprioceptive accuracy, but not for enhancement of knee repositioning, without sonification (Ghai et al., 2018), as well as immediate and delayed retention of decreased errors in stroke survivors (Fujii et al., 2016), and retention after a 1-month follow up after improving symmetrical standing (Wannsted & Herman, 1978). Retention was equally present for continuous and trajectory triggers, which both are thought to impact implicit learning processes, with trajectory triggers also utilizing explicit feedback, but no results are reported for exclusively explicit learning settings.
PMs: Non-Motor Outcomes
Overview of Motor and Non-motor Findings for Musically Informed Sonification (MIs) Systems, Grouped by Trigger
Note. HP = Healthy participants, VI = visually impaired, Nov = novices, Pro = professionals, TL = top level, AT = Athletes, SH = shooters, CS = championship, SP = Stroke patients, MS = multiple sclerosis, Amp = Amputees, Exp = experiments, ➢ = motor outcomes, ○ = subjective outcomes, ■ = retention outcomes, ↑ = significant increase, ↑* = trend toward significant increase, ↓ = significant decrease, ✓ = retention, Ø = no significant difference/retention.
Musically Informed Sonification (MIs)
Musically informed sonification (MIs) provides sounds that are generally discrete rather than continuous and include musical information to make the sound more musically coherent (e.g., scales, chords, melodies). For example, boat acceleration in rowing was translated into tones of a musical scale to represent rowing speed (Schaffert et al., 2011a; Schaffert & Mattes, 2015a, 2015b). An overview of all findings from MIs systems is provided in Table 4.
MIs: Movement Outcomes
As presented in Table 2, 35/128 (28%) of the included papers are categorized as MIs systems, of which 22/35 (63%) are compared to a control condition without sonification, and the remaining 13 systems are compared to other systems and are described below (Table 9). Table 4 lists the motor and non-motor outcomes of MIs systems compared to control, of which 20 (91%) show improvements (including trends), none show only degradations, 1 (5%) shows null results, and 1 (5%) are mixed.
For the motor outcomes, performance increases were observed in all studies (except one, see below), spanning both fine and gross motor skills. Specifically, for stroke survivors, this concerned mostly upper-limb function (Chen et al., 2016; Loria et al., 2022; Nikmaram et al., 2019; Raglio et al., 2021; Scholtz et al., 2015, 2016), and also gait (Owaki et al., 2021), which was also found for specific stepping sequences in people with Multiple Sclerosis (MS) (Moumdjian et al., 2022). For athletes, movement improvements include enhanced mean boat velocity, performance, time-structure of boat-acceleration curves, motor control stability, distance, but not stroke rate and acceleration values in rowing experiments (Schaffert et al., 2011a, Schaffert et al., 2011b; Schaffert & Mattes, 2014, 2015a, 2015b), technique while executing front flips (Levine et al., 2019), and peak positive acceleration in walking (Wood & Kipp, 2014). For upper-limb movements in healthy non-athletes, improved synchronization of a hand-held controller (Dotov & Froese, 2018), enhanced wrist movement learning (Ronsse et al., 2011), and effects on quantity, timing and stability of movements (Newbold et al., 2016, 2020) were found. Lastly, Oh et al. (2015) found no differences in swipe length and movement speed when making gestures using the fingertips.
Retention-related findings on rowing included increased mean boat velocity, enhanced motor control stability, increased performance over 500m distance and maintenance of improved technique in rowing was found following training sessions, with immediate and 24h retention of benefits (Schaffert & Mattes, 2014). Lastly, results reported by Wood and Kipp (2014) showed immediate retention of improved walking performance. Of the positive retention results, two use event-based and two use continuous triggers, showing positive results for both implicit and explicit feedback.
MIs: Non-Motor Outcomes
Subjective ratings of intuitiveness indicated that MIs was intuitive and (specifically for rowing) translated boat acceleration very clearly auditorily, increased specific movement awareness, attention, kinesthesia, and consciousness, as well as improved crew coordination (Schaffert et al., 2011a, Schaffert et al., 2011b; Schaffert & Mattes, 2014, 2015a, 2015b). In cycling, increased attention and movement awareness was reported, and MIs was again rated as intuitive (Schaffert et al., 2017). Additionally, stroke survivors showed enthusiasm during training with sonification (Nikmaram et al., 2019), and reported positive influences on reward, confidence, informativeness, and motivation (Newbold et al., 2016, 2020). Next to this, improved scores were reported for several validated movement-related questionnaires (Raglio et al., 2021; Scholtz et al., 2015), as well as reduced joint pain (Raglio et al., 2021).
Single Tone Sonification (STs)
Overview of Motor and Non-motor Findings for Single Tone Sonification (STs) Systems, Grouped by Trigger
Note. HP = Healthy participants, Nov = novices, Pro = professionals, AT = Athletes, SP = Stroke patients, CAI = chronic ankle instability patients, HC = handicapped, Exp = experiments, ➢ = motor outcomes, ○ = subjective outcomes, ■ = retention outcomes, ↑ = significant increase, ↓ = significant decrease, ✓ = retention, Ø = no significant difference/retention.
STs: Movement Outcomes
As presented in Table 2, 22/128 (17%) of the included papers are categorized as STs systems, of which 19/23 (87%) are compared to a control condition without sonification, and the remaining 3 systems are compared to other systems, and are described below (Table 9). Table 5 lists the motor and non-motor outcomes of STs systems, of which 14 (75%) show only improvements, none show only degradations, 1 (5%) shows only null results, and 4 (20%) are mixed. For the motor outcomes, increases were found in all studies but one, with the majority of studies involving walking or posture. Various improved gait or balance parameters were reported for stroke survivors (Jung et al., 2019; Kim et al., 2021) and for healthy participants (An et al., 2019; Cha et al., 2018; Donovan et al., 2016; Ferrigno et al., 2016; Hasegawa et al., 2017; Ki et al., 2015; Torp et al., 2021, 2022), but not in lateral hop (Torp et al., 2021) or in walking speed, stride length nor cadence (Ferrigno et al., 2016). Positive effects were found for stroke survivors (Thielman, 2010) as well as healthy participants (Boyer et al., 2013; Rand, 2018) for most motor outcomes, with the exception of outcomes of shoulder flexion, elbow extension, grip strength (Thielman, 2010), as well as in directional accuracy when pointing at auditory targets (Boyer et al., 2013). Improvements in sports included technique swimming (Chollet et al., 1979), cycling (O’Brien et al., 2020), golf putting (Simek & O'Brien, 1978), and weight bearing training (Vijittrakarnrung et al., 2020).
In terms of retention, one study examined this with an error-based trigger, eliciting explicit learning strategy, and reported retention of benefits after learning in a sonification context (Torp et al., 2022).
STs: Non-Motor Outcomes
For STs, the search yielded no studies that reported ratings of intuitiveness, of motivation-related outcomes or of individual differences.
Rhythmic Sonification (Rs)
Overview of Motor and Non-motor Findings for Rhythmic Sonification (Rs) Systems, Grouped by Trigger
Note. HP = Healthy participants, SP = Stroke patients, CP = cerebral palsy patients, MS = multiple sclerosis patients, PD = Parkinson patients, ➢ = motor outcomes, ○ = subjective outcomes, ■ = retention outcomes, ↑ = significant increase, ↓ = significant decrease, ✓ = retention, Ø = no significant difference/retention.
Rs: Movement Outcomes
As presented in Table 2, 17/130 (13%) of the included papers are categorized as Rs systems, of which 9/17 (53%) are compared to a control condition without sonification, and the remaining 8 systems are compared to other systems, and are described below (Table 9). Table 6 lists the motor and non-motor outcomes of Rs systems compared to control condition, of which 8 (88%) show only improvements, none show only degradations or null results, and 1 (12%) is mixed, and most concern walking or standing. Gait improvements were mostly reported for clinical populations such as stroke survivors (Bang, 2016; Yang et al., 2016), people with cerebral palsy (Baram and Lenger, 2012), Parkinson’s disease (Baram et al., 2016), and MS (Baram and Miller, 2017), but also for healthy participants (Reh et al., 2022), with one null result found for quality of standing up movement in stroke (Bang, 2016). For upper limb movement, increases in performance were found in healthy participants (Chiou & Chang, 2016; van Vugt & Tillmann, 2015) and for stroke survivors (Bang, 2016; Secoli et al., 2011). For squat movements, enhanced balance and reduced errors were found (Hale et al., 2020). In terms of retention, both Van Vugt & Tillman (2015) and Chiou and Chang (2016) show retention after feedback removal, and for clinical groups, Baram et al. (2016; Baram & Miller, 2017) show retention of gait benefits at 10 and 15 minutes, Secoli et al. (2011) show retention of arm movement effects, and Hale et al. (2020) show retention of squat movements. All but the latter 2 use event-based triggers, utilizing explicit learning strategies, while the remaining 2 studies used a trajectory-based trigger, combining implicit and explicit learning.
Rs: Non-Motor Outcomes
For Rs, the search yielded no studies that reported ratings of intuitiveness, of motivation-related outcomes or of individual differences.
Music-Based Sonification (MBs)
Overview of Motor and Non-motor Findings for Music-Based Sonification (MBs) Systems, Grouped by Trigger
Note. HP = Healthy participants, Pro = professionals, PEB = physical education background, AT = Athletes, SP = Stroke patients, ➢ = motor outcomes, ○ = subjective outcomes, ■ = retention outcomes, ↑ = significant increase, ↓ = significant decrease, Ø = no significant difference.
MBs: Movement Outcomes
As presented in Table 2, 13/129 (10%) of the included papers are categorized as MBs systems, of which 7/13 (54%) are compared to a control condition without sonification, and the remaining 6 systems are compared to other systems, and are described below (Table 9). Table 7 lists the motor and non-motor outcomes of MBs systems compared to a control, of which 4 (66%) show only improvements, none show only degradations or null results, and 3 (43%) are mixed. In athletes, improvements in running indices are found (Lorenzoni et al., 2018; van der Berghe et al., 2021, 2022), with no differences found for cadence (van der Berghe et al., 2021), as well as improvements in deadlifts (Lorenzoni et al., 2019), nor in effectiveness for cycling (van der Vlist et al., 2011). Upper-limb findings show reduced errors in tracking in healthy participants (Dailly et al., 2012) and reaching in stroke survivors (Douglass-Kirk et al., 2023). None of the included studies involving MBs reported outcomes on learning retention.
MBs: Non-Motor Outcomes
Lorenzoni et al. (2019) report that subjective participant ratings on perceived effort, clarity of feedback, pleasantness, motivation, accuracy, intuitiveness and usability were not affected by sonification. However, positive effects of MBs on fun and enjoyment, perceived competence, attentional focus, distress, as well as its value and usefulness have been subjectively reported, but not on perceived exertion (van der Vlist et al., 2011). No results on individual differences were reported.
Environmental Sonification (Es)
Overview of Motor and Non-motor Findings for Environmental Sonification (Es) Systems, Grouped by Trigger and Subtype
Note. HP = Healthy participants, Nov = novices, Pro = professionals, PI = physically inactive, AT = Athletes, HAP = hip arthroplasty patients, Exp = experiments, ➢ = motor outcomes, ○ = subjective outcomes, ■ = retention outcomes, ↑ = significant increase, ↓ = significant decrease, Ø = no significant difference.
Es: Movement Outcomes
Overview of Sonification Findings Comparing Types, Grouped by Comparison and Trigger
Note. HP = Healthy participants, Nov = novices, Pro = professionals, AT = Athletes, HAP = hip arthroplasty patients, MISJ = mobility issues of shoulder joint patients, Exp = experiments, pp = Participants, ➢ = motor outcomes, ○ = subjective outcomes, ■ = retention outcomes, ↑ = significant increase, ↑* = trend toward significant increase, ↓ = significant decrease, ✓ = retention, Ø = no significant difference/retention, & = both types, > = outperforming.
Es: Non-Motor Outcomes
No findings are reported on the intuitiveness of these system, but one experiment showed positive subjective participant ratings on walking with Es (wind, water, or can-crush) sounds, with effects on feelings of body weight, tiredness, having control, and being more comfortable, motivated, and happier (Ley-Flores et al., 2019), and an engaging and pleasant experience (Pugliese & Takala, 2015). However, delayed Es sounds (as compared to non-delayed) are rated as disturbing (Kennel et al., 2015), and one study found no effect on emotional attention while walking (Turchet & Bresin, 2015). No findings on individual differences were reported.
Evaluating Outcomes Between Sonification Types
Several studies (21/101, 21%) made direct comparisons between sonification types, shown in Table 9. Various types of systems were compared to each other rather than to a control condition without sonification. As such, these comparisons do not test the effect of sonification but rather the usefulness between particular types.
Movement Outcomes Between Sonification Types
Table 9 provides details on 51/130 (39%) systems that are compared to each other, listing their motor and non-motor outcomes. PMs systems were compared to MIs, Rs, STs, and Es systems, overall showing mixed results where PMs improved performance more than the other sonification system in 3/15 comparisons (Fehse et al., 2020; Fuchs et al., 2020; O’Brien et al., 2020b), showed worse performance in 3/15 comparisons (Hummel et al., 2010; Vidal et al., 2020; Zanotto et al., 2013) and showed null results in the remaining 9/15 comparisons (Fehse et al., 2020; Horsak et al., 2016; Hummel et al., 2010; O’Brien et al., 2020b; Reh et al., 2021; Vidal et al., 2020). A clear pattern in terms of the comparison system does not emerge. MIs systems were compared to PMs, Rs, MBs, and Es systems, overall showing mixed results where MIs improved performance more than the other sonification system in 4/16 comparisons (Dyer et al., 2017; Hummel et al., 2010; Singh et al., 2015), showed worse performance in 4/16 comparisons (Fehse et al., 2020; Fuchs et al., 2020; Liu et al., 2022; Maes et al., 2019) and showed null results in the remaining 8/16 comparisons (Fehse et al., 2020; Horsak et al., 2016; Hummel et al., 2010; Maes et al., 2019; Reh et al., 2021; Varni et al., 2012). Again, a clear pattern in terms of the comparison system does not emerge. STs systems were compared to PMs and Es systems, overall showing mixed results where STs improved performance more than the other sonification system in 1/4 comparisons (Vidal et al., 2020), showed worse performance in 1/4 comparisons (Ley-Flores et al., 2021) and showed null results in the remaining 2/4 comparisons (Dubus & Bresin, 2015). Here, the results were not only mixed but there were too few comparison systems to discern a pattern. Rs systems were compared to PMs, MIs, STs, MBs, and Es systems, showing somewhat mixed results where Rs improved performance more than the other sonification system in 3/8 comparisons (Maes et al., 2019; Zanotto et al., 2013), showed worse performance in 1/8 comparisons (Dyer et al., 2017) and showed null results in the remaining 4/8 comparisons (Dubus & Bresin, 2015; Maes et al., 2019; O’Brien et al., 2020b). Here, positive results are a bit more prominent but as all comparisons concern different types, no generalizations can be made. Next, MBs systems were compared to PMs, MIs, MBs, and Es systems, generally showing null or negative results, where MBs improved performance more than the other sonification system in 1/8 comparisons (Liu et al., 2022), showed worse performance in 2/8 comparisons (Maes et al., 2019; Singh et al., 2015) and yielded null results in the remaining 5/8 comparisons (Horsak et al., 2016; Maes et al., 2019; Singh et al., 2015; Varni et al., 2012). Here the negative results were found comparing to MIs and Rs. Finally, Es systems were compared to PMs, MIs, STs, Rs, and MBs systems, again showing somewhat mixed results where Es improved performance more than the other sonification system in 1/6 comparisons (Ley-Flores et al., 2021), showed worse performance in 2/6 comparisons (O’ Brien et al., 2020b; Singh et al., 2015) and showed null results in the remaining 6/6 comparisons (Dubus & Bresin, 2015; O’ Brien et al., 2020b; Reh et al., 2021; Singh et al., 2015). Importantly, we here include each comparison from each reference point, leading to double reporting of each finding.
In terms of retention, comparisons between systems are made by Zanotto et al. (2013) who show better retention for Rs than for PMs, when each comparing to control. Dyer et al. (2017) show that using MIs allows the recovery of declined learning after 24 hours, by playing the sound of a perfect movement execution, which Rs does not.
Non-Motor Outcomes Between Sonification Types
Aspects of intuitiveness were rated in 19 systems, and contrasted in six comparisons, involving PMs (6), MIs (5), STs (1), Rs (4), MBs (2) and Es (1). High levels of intuitiveness were found for PMs, MIs, Rs, and STs (Brückner et al., 2012; Kirby, 2009; Schaffert, Engel, et al., 2019). Comparing to each other, PMs was rated more intuitive or understandable than both MIs and Rs (Brückner et al., 2012; Connor et al., 2022) and once less intuitive than MIs (Schaffert, Engel, et al., 2019), specifically for movement timing in a rhythmic movement. PMs and Rs were rated as more intuitive than MIs (Fuchs et al., 2020), as well as compared to MIs, MBs, and Es (Bevilacqua et al., 2018). Finally, MBs was rated as more intuitive than MIs (Liu et al., 2022). Overall, this suggests that PMs and Rs are generally rated as the most intuitive, and MIs, MBs, and Es less so.
For motivational value, aspects of motivation or pleasure were rated in 10 systems, and contrasted in two comparisons, involving PMs (2), MIs (4), Rs (1), MBs (2) and Es (1). High levels of preference or motivation were found for MIs, MBs, and Es (Singh et al. (2015). Comparing to each other, MIs, as well as MBs and Es, were found to be more motivating or pleasurable than PMs and Rs (Bevilacqua et al., 2018; Connor et al., 2022).
Two results on individual differences were reported, comparing musicians and non-musicians, showing mixed results. Specifically, Varni et al. (2012) show that expert level participants preferred no sonification over both PMs and MIs, and Liu et al. (2022) show that musicians show movement improvement when using Mis while non-musicians do not.
Discussion
The present systematic review aimed to firstly create a taxonomy of sonification systems by categorizing them by type of sound and how it is triggered. In addition, it synthesizes findings on motor and non-motor aspects of performance reported for each of these sonification types, while also considering their triggers in terms of learning retention. As hypothesized, each sonification type is shown to be effective in improving various movement-related performance outcomes, and while some types were more often used for specific movements (i.e., gait in STs and Rs), overall, a wide range of movements and participant groups showed positive effects of sonification. Moreover, retention of benefits after removing sonification was found for a mix of explicit and implicit learning strategies. However, the effect on participants’ subjective ratings of intuitiveness and motivation appear to differ between types, partly confirming our hypotheses. Lastly, findings on the influence of personal characteristics were considered, but are too scarce for interpretation.
We evaluated 130 auditory movement sonification systems, as described by existing literature. In the process of the review, we conceptualized six sonification types (PMs, MIs, STs, Rs, MBs, and Es), and four different sonification triggers (error-based, event-based, continuous, and trajectory). Notably, our results indicate that PMs and MIs systems, which are the most common, mainly are coupled with continuous, and to a lesser extent trajectory triggers, thus generally utilizing implicit learning strategies. Conversely, STs, Rs, and Es mainly use error- and event-based triggers, leading to explicit learning strategies, and MBs show a mixed use of learning strategies, most often using trajectory triggers. These types and triggers offer opportunities to interpret the findings in a more fine-grained manner in future work.
As expected
For periodic movements, MIs, STs, Rs, and MBs, but not PMs and Es, appear to be primarily suitable to improve performance. For the former, existing literature describes movement timing-related improvements in gait (Owaki et al., 2021; Wood & Kipp, 2014), synchronization in rowing (Schaffert et al., 2011a, Schaffert et al., 2011b; Schaffert & Mattes, 2014, 2015a, 2015b), bimanual arm elevation, and hand movements, including finger tapping (Dotov & Froese, 2018; Newbold et al., 2016, 2020; Ronsse et al., 2011). Interestingly, MIs outperformed Rs in acquisition of a bimanual shape-tracking task (Dyer et al., 2017), and while Es was used with gait, the outcomes were mixed and also included perturbations (Kennel et al., 2015; Reh et al., 2019). Taken together, we argue that sonification types that include rhythm (MIs, STs, Rs, and MBs) may be most helpful in adjusting movement periodicity-related aspects, as compared to other sonification systems.
Concerning learning retention, explicit learning approaches were expected to more often affect retention positively as compared to implicit learning approaches. Implicit approaches, represented by continuous or trajectory trigger (where the latter also includes an explicit feedback element) comprised 2/3 of all systems, however studies examining learning retention were scarce. For the studies reporting outcomes of retention, a mixed selection of systems using implicit and explicit learning strategies was found, and while retention was always found for triggers using explicit strategies, mixed results were found for triggers with only implicit strategy use, suggesting that explicit strategies more reliably lead to retention of learned motor skills. This mostly confirms our expectation, but it should be noted that the results on learning retention were limited, and further investigation of these mechanisms for specific movement types is warranted.
With regards to non-motor measures, positive ratings of intuitiveness, indicating how easy the system was to understand, were most often reported for PMs, Rs, and MIs, respectively, but not for MBs, Es and STs. As compared to a no-sound condition, both PMs and MIs have been rated as highly intuitive (Schaffert et al., 2011a; Stienstra et al., 2011). Studies comparing between types indicate that both PMs and Rs are generally rated as more intuitive when compared to MIs (Brückner et al., 2012; Fuchs et al., 2020), but not as interesting or motivating to listen to (Bevilacqua et al., 2018; Connor et al., 2022). Conversely, Schaffert, Engel, et al. (2019) show that expert swimmers rated MIs as more intuitive than PMs, but this was specifically related to the rhythm of movements being represented by the rhythm of the individual tones, as compared to continuous pitch. This is in line with Hermann et al. (2011)’s suggestion that musically informed sounds are preferable due to the ease of perceiving musical tones, opposed to continuous pitch, indicating movement-specific characteristics for intuitiveness of sound. As such, we argue that MIs may be rated as more intuitive in the specific case where the rhythm of executed movements is important, but in most cases, PMs and Rs types are rated as more intuitive and understandable than MIs systems.
In terms of motivation, movement sonification seems to generally be effective in increasing autonomous motivation, as compared to executing the same movements without sonification. Specifically for PMs, motivation-related results show that participants reported increased engagement and pleasantness (Pugliese & Takala, 2015), as well as enjoyment (Yang & Hunt, 2015). Furthermore, MIs may positively affect perceived reward, amount of movement, and motivation, among others (Newbold et al., 2016, 2020), with highly enthusiastic responses from stroke survivors (Nikmaram et al., 2019). Moreover, MIs was associated with reduced joint pain ratings (Raglio et al., 2021; Scholtz et al., 2015a; Scholtz et al., 2016). Next to this, positive effects of MBs on fun and enjoyment, perceived competence, attentional focus, distress, as well as its value and usefulness have been subjectively reported (van der Vlist et al., 2011). Lastly, Es reportedly seems to influence perceived body weight, feelings of tiredness and having control, as well as being more comfortable, motivated, and happier (Ley-Flores et al., 2019). Notably, all these results are mainly evaluated in comparison to a no sound control condition, with a few exceptions indicating that MIs was rated as more interesting and motivating to listen to than both Rs and PMs (Bevilacqua et al., 2018; Connor et al., 2022).
Although we aimed to also address the role of individual differences in the efficacy of sonification systems, this was mostly neglected in the included articles, such that no reliable conclusions could be drawn. Two articles that did consider such differences both focused on musicianship, but showed mixed results (Liu et al., 2022; Varni et al., 2012). While this suggests that individual differences, and especially existing skill level, may be of importance when evaluating sonification systems, future work should address this issue.
Conclusion
We categorized existing movement sonification systems into six types of used sounds, (PMs, MIs, STs, Rs, MBs, and Es) and four triggers (event-, error-, trajectory-based, and continuous) that initiated these sounds. Adopting this terminology in future research may enhance clarity and facilitate the comparison of system outcomes to further refine our understanding of their effects. Over all systems, sonification was overwhelmingly shown to have positive effects on movement aspects. Findings suggest that PMs, and to a lesser extent MIs, but not Rs, MBs, Es and STs, are useful to adjust subtle movement aspects, likely based on the directness of movement-sound coupling techniques, use of implicit learning approaches, and mapping multiple movement-sound parameters. Fitting with their sound type, sonification systems using rhythmic sounds (MIs, STs, Rs and MBs) appear most suitable to adjust periodic movement. In general, users rate PMs and Rs as more intuitive than other sonification types. In contrast, MIs is generally liked, or increases autonomous motivation, more than both PMs and Rs. Overall, movement sonification increases motivation, as compared to no-sound control conditions. Retention of the effects of sonification-based learning may be more related to explicit rather than implicit learning approaches.
For clinical settings, MIs sonification systems would be recommended as the most suitable for rehabilitation, as motivation is of great importance in these settings. Conversely, PMs may provide more informative feedback, affording more independent used of the system, implicitly informing the user about subtle movement aspects, potentially allowing for more specific learning. Future research may evaluate performance of either independent systems or between sonification types or triggers (explicit vs implicit), and include experimental conditions based on level of individual participants' expertise, long-term training, as well as long-term retention tests, while also considering individual differences. However, based on the literature reviewed here, it is clear that sonification offers very promising avenues for movement improvement in both clinical and non-clinical settings.
Supplemental Material
Supplemental Material - Movement Sonification Types and Triggers: A Systematic Review
Supplemental Material for Movement Sonification Types and Triggers: A Systematic Review by Marijn Coers, Maria M. Petratza, Alessandro Palumbo, Marit F. L. Ruitenberg, Ineke J. M. van der Ham, Rebecca S. Schaefer in Perceptual and Motor Skills
Supplemental Material
Supplemental Material - Movement Sonification Types and Triggers: A Systematic Review
Supplemental Material for Movement Sonification Types and Triggers: A Systematic Review by Marijn Coers, Maria M. Petratza, Alessandro Palumbo, Marit F. L. Ruitenberg, Ineke J. M. van der Ham, Rebecca S. Schaefer in Perceptual and Motor Skills
Footnotes
Acknowledgements
We would like to thank Marianne Bos, who provided input at the beginning of the conceptualization of this review, and Jan Schoones, who provided crucial help with the formulation of search terms their adaptation to difference data bases. MC is supported by a grant to RSS from the Dutch Scientific Organization NWO (Aspasia grant nr 015.015.017).
ORCID iDs
Funding
The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: MC is supported by a grant to RSS from the Dutch Scientific Organization NWO (Aspasia grant nr 015.015.017).
Declaration of Conflicting Interest
The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Data Availability Statement
Data extraction table is available upon request.
Supplemental Material
Supplemental material for this article is available online.
Author Biographies
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
