Sage Journals: Discover world-class research

Abstract

The aim of this work is to redefine the concept of sensorimotor augmentation, extending it from wearable devices to supernumerary robotic upper limbs that are grounded and can be placed out of the physiological peripersonal space of the user. While the wearability of the supernumerary robotic limb (SRL) is not mandatory, the human-robot connection must be maintained through a wearable interface that allows the user to naturally command and receive sensory feedback from the SRL. In this way, users can keep the robot under their control, without adding unnecessary extra weight on their body. As a result, the application of SRLs in assistive contexts is facilitated, as even people with severe paresis who might not be able to withstand the burden of a wearable robotic arm, can be augmented by grounded SRLs. Focusing on upper-limb assistive augmentation, this paper introduces another fundamental novelty: the exploitation of augmentative technologies as motivational tools for re-using the impaired upper limb in chronic paretic patients. To this aim, a new user-centered procedure for extracting a control signal from a residual motion of a patient’s upper limb is proposed and tested in scenarios where the impaired limb takes part in the manipulation task by controlling the SRL. Extensive pilot tests with two post-stroke patients show the intuitiveness and usability of the proposed approach.

Keywords

augmentation supernumerary limbs sensorimotor interfaces assistive robotics

Introduction

This work introduces upper-limb sensorimotor augmentation technologies enabling co-manipulation tasks in assistive scenarios. Different robots and interfaces are tested in several daily-living bimanual tasks with post-stroke patients, outlining their main advantages and disadvantages in terms of usability and perceived workload. Considered robots include both wearable devices, whose base is fixed to the human body (supernumerary finger), and grounded devices, whose base is placed out of the human physiological peripersonal space (supernumerary gripper, supernumerary arm).

While wearable supernumerary limbs have now their established place in the field of augmentation (Prattichizzo et al., 2021), here we introduce a new concept of human-robot augmentation for the upper-limbs, where supernumerary robots, especially arms, are not necessarily worn on the user’s body, but can also be grounded. This idea enables the application of augmentative technologies in assistive scenarios, as people with disabilities may lack the physical strength or dexterity required to operate a wearable robotic limb. While the wearability of the supernumerary robotic limb (SRL) becomes not mandatory, the wearability of the interface is still key for the sensorimotor augmentation. Having a wearable sensorimotor interface guarantees the connection between the human and the SRL even when the latter is not wearable, allowing users to control the robot and receive feedback from it as if it was part of their own body. In this way, the focus can be shifted to the design and development of the interface, while the robot can be chosen according to the envisaged application and to the level of strength or dexterity required from it. A sketch of a possible scenario is shown in Figure 1.

Figure 1.

Concept: a person with a paretic upper limb performs an augmented manipulation task with a grounded supernumerary arm equipped with a supernumerary gripper. Wearable sensorimotor interfaces connect the user to the robot. Commands for the robot are generated by moving the paretic arm and by using push buttons embedded in a ring-shaped interface. Tactile interfaces worn on the healthy arm allow the user to be aware of the robot state.

The wearable sensorimotor interfaces developed in this work ensure a bidirectional connection between the human and the SRL exploiting continuous and event-driven control and feedback modalities (Prattichizzo et al., 2021). The user can command the robot using buttons, body motions, or a combination of the two, whereas the feedback is delivered through vibrotactile stimuli synchronized with the task.

Regarding the target population, this paper focuses on chronic post-stroke patients who have a paresis of their upper arm with residual motion, but are not able to apply grasping or pinch grip forces, or, even worse, have completely lost their grasping ability. Post-stroke patients frequently tend to use the healthy side to compensate for movements of the impaired side, leading to a non-use of the impaired limbs which exposes them to the risk of subsequent muscle atrophy (Dennler et al., 2023). According to different physiological cortico-motoneuronal controls, this tendency is more frequently observed for upper limbs than for lower limbs (Rand and Eng, 2012). The idea of this work is to exploit the augmentation system not only to let the users regain independence in certain daily activities, but also to motivate them to purposefully move their impaired arm. In other words, the robotic system should be regarded at the same time as an aid and a motivational tool. In the envisaged scenarios, the impaired limb is involved in the augmented manipulation task either because it supports the SRL, as in the case of the wearable supernumerary finger, or as its motion is coupled to that of the SRL, as in the case of the supernumerary gripper and of the supernumerary arm. The coupling is achieved thanks to a new user-centered strategy that allows to extract an effective control signal from the user’s specific residual motion and then map it to the robot actuation system.

The rest of the paper is organized as follows. In the second section, the state of the art of human-robot augmentation is summarized, describing how this work goes beyond it. In the third section, the newly introduced user-centered method for extracting a control signal from the residual motion of impaired upper-limbs is detailed, along with the implemented control and feedback modalities. The experimental protocol is described in the fourth section, whereas the results in terms of objective and subjective evaluations are presented in the fifth and sixth sections, respectively. A summary of the key findings is given in the seventh section, whereas the conclusions and future directions of the work are outlined in the last section of the paper.

Related works

Supernumerary robotic limbs are robots designed to augment the sensorimotor capabilities of humans (Prattichizzo et al., 2021). Differently from prostheses, which substitute missing limbs, and exoskeletons, which reinforce existing ones, SRLs add extra degrees of freedom to the human body.

Application and design of SRLs

SRLs find applications in assistive scenarios, to support people with motor disabilities (Ciullo et al., 2020; Hasanen et al., 2024; Salvietti et al., 2016), in industrial and service tasks, to enhance workers’ physical abilities (Ballesteros et al., 2023; Huo et al., 2023; Tu et al., 2023), and in entertainment and art, to foster new forms of creativity and social interaction (Inami, 2023).

In the current sense, SRLs are considered to be wearable devices; therefore, their design has to satisfy stringent safety and ergonomics requirements. While supernumerary fingers are expected to have limited functions, that is, mainly grasping, and thus can have a rather compact design (Hussain et al., 2017), wearabilty is critical in supernumerary hands and arms, which are meant to perform more complex operations and thus should feature several actuated degrees of freedom (DoFs), potentially adding a lot of extra weight to the wearer’s body.

Only a notable example of supernumerary hand is present in the literature. It has been applied to support patients with upper-limb disabilities, but, to be used, it requires external assistance to support the weight of the user’s paretic arm (Ciullo et al., 2020; Trompetto et al., 2022).

State-of-the-art solutions for supernumerary arms either adopt commercial lightweight robots attached to purposefully designed backpacks (Khoramshahi et al., 2023), or propose the complete design of the system including the arm and the equipment to wear it (Du et al., 2023; Saberpour Abadian et al., 2023). Recent advancements in the design of supernumerary arms include the use of soft actuators and materials, to ensure the intrinsic safety of the structure (Liu et al., 2020; Xu et al., 2024). Apart from a few examples in which assistive applications are envisaged but not tested with patients (Hasanen et al., 2022, 2024), previous works on supernumerary robotic arms mainly consider wearable devices thought for healthy people.

Tight design constraints in terms of portability and compactness also apply for the well-established field of wheelchair-mounted manipulators (Jaeger et al., 2023; Maheu et al., 2011). However, these solutions address a smaller target population compared to the broader group of potential users who might benefit from a supernumerary arm but do not rely on a wheelchair.

Control of SRLs

The control of SRLs is closely related to their design, as an increasing number of DoFs of the SRL leads to a larger configuration space for the robot, which requires the adoption of high-level control strategies for coordinating the motion of multiple DoFs (Prattichizzo et al., 2014; Takashita et al., 2024). A trade-off must be found between the user’s direct control and the level of autonomy of the robot. In this regards, algorithms for recognizing, predicting, and autonomously compensating for human motions have been proposed (Luo et al., 2021; Verdel et al., 2024; Zhang et al., 2023). SRLs are meant to work under the direct control of the user, who should learn how to coordinate their motion with that of the biological limbs. This coordination can be achieved by reallocating neural resources not directly involved in the task to control the robotic limbs without interfering with the control of the natural ones (Dominijanni et al., 2021). For example, musculoskeletal redundancy can be exploited to generate control signals for the robot. This can be achieved using control interfaces based on surface electromyography or on wearable or grounded sensors for motion capture to detect user’s body motions (Lee et al., 2024; Lisini Baldi et al., 2024).

Since SRLs operate in close proximity with the user, vision is the primary sense that allows perceiving the robot state and the task progress. However, additional feedback strategies can enhance user’s control of the SRL reinforcing or complementing visual cues. When the SRL is worn on the user’s body, it generates an inherent haptic feedback that is helpful in certain tasks (Guggenheim and Asada, 2020), but might not be enough when precise operations (e.g., force regulation) have to be performed (Buratti et al., 2023), or when the robot state is difficult to infer through vision (e.g., in case of occlusions or delays) (Pinardi et al., 2023b). In these cases, providing additional feedback (e.g., vibrotactile stimuli) can make the user more aware of the situation (Hussain et al., 2017; Pinardi et al., 2023a; Prattichizzo et al., 2021).

Contribution

This work proposes integrated augmentation systems for wearable and grounded SRLs. While wearable supernumerary fingers are now an established technology in the field, relaxing the constraint of wearability for supernumerary grippers and arms allows broadening their potential applications to contexts where wearability would be infeasible, as in the assistance to patients with upper-limb impairments in daily activities. In the proposed paradigms, while the physical connection with the robot is lost, the user retains control over it thanks to wearable sensorimotor interfaces.

The proposed concept of sensorimotor augmentation (Figure 1) merges the advantages of human-robot collaboration, where the robot allows to perform tasks that go beyond biological capabilities (Ajoudani et al., 2018), with the distinctive features of advanced robotic teleoperation, where there is a bilateral connection between the user and the robot (Torielli et al., 2024). Different from previous works on close-proximity teleoperation (Baldi et al., 2017; Poignant et al., 2024), however, the proposed control interface solely exploits the residual motion of the impaired upper limb without involving any body part that would not naturally take part to the task (e.g., head). As a result, a body part that would be involved in the task for healthy individuals is here repurposed to control the SRL.

This approach is particularly relevant for post-stroke patients, who are the target of several rehabilitation robotics technologies proposed in the literature (Reis et al., 2021), but are rarely involved in applications with assistive robotic manipulators. The latter usually target people with very limited motor abilities, who need complete assistance. Thus, current works in assistive robotics focus on making the robots as autonomous as possible (Goldau et al., 2019; Jenamani et al., 2024; Zhang and Demiris, 2022), with the human input usually given through non-wearable interfaces or with vocal commands when needed (Banerjee et al., 2024; Bhattacharjee et al., 2020), and without any feedback from the robot to the user (Nanavati et al., 2023). In this paper, instead, the supernumerary limbs augment users’ capabilities and interact physically with the human limbs, while being under users’ direct sensorimotor control thanks to wearable interfaces that also provide tactile feedback. Differently from previous works on physical assistive robotics (Nanavati et al., 2023), the main focus here is on bimanual operations, which are difficult to perform after a stroke (Frennert et al., 2024).

In this work, extensive experimental trials with two post-stroke patients have been conducted to show how the distinct features of the adopted SRLs can be leveraged in bimanual tasks through different control interfaces and strategies.

Methodology

Augmentation systems: Overview

The proposed augmentation systems have different levels of complexity and consist of a 1-DoF wearable supernumerary finger, a 2-DoF portable supernumerary gripper, and a grounded supernumerary arm equipped with the supernumerary gripper. These can be controlled using two different wearable input interfaces, one consisting of ring-mounted push buttons, designed to be worn on the healthy hand, and one based on user’s body motions, intended for the impaired limb. For the latter, a new user-centered procedure to extract a control signal from the patient’s residual motion has been implemented (Figure 2, Figure 3). The supernumerary limbs can either be commanded in an event-driven mode or in a continuous way, and each control method has a corresponding feedback strategy based on vibration patterns delivered either to the finger or to the arm (Figure 4). In the envisaged scenarios, the user controls a single task parameter at a time. However, to enhance the flexibility of the proposed systems and ensure that even rather complex tasks can be performed, strategies allowing the user to switch between the different degrees of freedom of a single robot or between different robots have been devised (Figure 4).

Figure 2.

Process to extract the control signal from a chosen motion of the impaired arm. In the calibration phase, tracking data (i.e., the relative rotations) describing repeated motions are analyzed to obtain the main direction and the range of motion. In the online computation, the online tracking data are projected and scaled according to the results of the calibration to obtain the control signal.

Figure 3.

(a) IMU placement on the patient’s impaired arm and (b) control signal and corresponding body motions. In this representative case, the patient chose as the control motion to raise and lower the arm while keeping the elbow closed. The corresponding control signal is an adimensional value in the range between 0 and 1.

Figure 4.

Block schemes illustrating the implemented control strategies and the use of the two different control interfaces. The orange color marks the user’s control inputs, whereas the gray areas indicate the blocks in which the control input is processed and mapped to predefined robot actions (event-driven control—EDC), continuous motions (continuous control—CC), or to the activation/deactivation of controllable DoFs. Concerning the scheme at the bottom, in our implementation, the activated DoF can then be controlled using body motions either in EDC or CC modality.

Extracting a control signal from residual motions

The proposed process aims at finding a control signal that is both practical for the patient and compatible with the control of the SRL in terms of reproducibility and robustness. As depicted in Figure 2, the process consists of two main steps: calibration and online computation. Examples of how the tracking data are processed are reported in the Supplemental Material. In both phases, wearable inertial measurement units (IMUs) are used to track the participant’s arm. The sensors are positioned as shown in Figure 3(a) to ensure precise tracking of the user’s arm. The IMUs orientations expressed in quaternion form are used to calculate the relative rotations between the joints based on the human arm kinematic structure. The resulting tracking data thus consist of the rotations of the arm relative to the chest, the forearm relative to the arm, and the wrist relative to the forearm, in quaternion form.

Before initiating the calibration process, the user is instructed to choose a motion of the impaired arm that is reproducible and not too tiring, especially when repeated multiple times. This motion must be comfortable for the participant and can involve any type of arm movement, based on individual preferences and residual capabilities.

Once the motion is selected, the calibration process begins. The participant is asked to repeat the chosen motion 10 times while tracking data are collected and stored. These data are then processed offline using principal component analysis (PCA) to extract a direction of motion that is sufficiently descriptive of the user’s motion. The resulting direction is then used in the online phase to project the multidimensional space of relative rotations onto a one-dimensional control signal. In particular, the first principal component is selected as the main direction of motion if it explains at least 80% of the total data variance. Otherwise, the calibration is repeated choosing a different motion. Once the main direction is determined, the calibration tracking data are projected onto the main direction of motion to find local maxima and minima. The range of motion was calculated as the difference between the mean of the local maxima and the mean of the local minima.

In the online computation phase, the real-time tracking data gathered from the motion of the arm are processed using the parameters obtained during the calibration. Specifically, they are projected onto the main direction of motion and normalized between 0 and 1 using the extreme values of the range of motion. An example of control signal is shown in Figure 3(b).

Event-driven control

In the event-driven control (EDC), the user’s input triggers a predefined action of the supernumerary limb through a finite state machine with three states (Figure 4-top). For the opening/closing of the finger or of the gripper, the first input closes the device, the second stops its motion, and the third activates its opening. For the supernumerary arm, the first input starts the forward motion of the arm along a predefined trajectory with a fixed velocity, the second stops the robot, and the third initiates the backward motion. The order of the actions is always the same, while the input changes depending on the control interface. For the ring interface, pressing the button provides input to the state machine. For the body motion interface, a trigger is generated when the control signal overcomes a predefined threshold.

In terms of feedback, when the control is based on buttons, vibrotactile cues are provided to the user through the same ring interface which embeds the buttons, whereas when the control is based on body motions, a tactile armband is used to deliver vibrations. In both cases, the number of vibration bursts delivered by the tactile interface depends on the state of the supernumerary limb. The specific mapping adopted in this work is detailed in Section “Experimental setups”.

Continuous control

In the continuous control (CC), the input provided by the user is continuously mapped onto a control signal for the robot. Two different strategies have been developed depending on the adopted supernumerary limb: position control and velocity control. The first is chosen for robots having a limited range of motion (finger, gripper), whereas the second is applied to the supernumerary arm (Figure 4-middle).

To control the opening/closing of the finger or of the gripper, the user can either keep a predefined button pressed (one for opening, one for closing), or use the selected motion of the impaired arm. In the first case, the release of the button or the reaching of the motor limits stop the robot motion. In the second case, the range of motion of the user is directly mapped onto the range of motion of the fingers.

A velocity control strategy has been implemented for the supernumerary arm, as position control would have been less robust to possible delays in the communication, resulting in a less smooth robot motion. The user can control the velocity of the robot end-effector by moving the impaired arm. The participant’s range of motion is mapped so that when the user keeps their arm in an interval around the mid-range position, the robot stops. Then, if the user goes above the mid-range, the robot goes forward, while moving below the mid-range causes the robot to move backward. In both directions, the velocity reaches the predefined maximum value when the user arrives to the extremes of the trajectory.

As in the event-driven control, each control interface is related to a certain feedback interface: the ring is used when the control is based on buttons, whereas the tactile armband is adopted when the control is based on body motions. In both cases, a certain number of vibration bursts is sent depending on the value of the control signal. The specific strategy herein adopted is detailed in Section “Experimental setups”.

Switching between multiple degrees of freedom

To enable the control of multiple degrees of freedom a combination of the two proposed input interfaces is used. In particular, buttons are used to select the DoF to be controlled (Figure 4-bottom), whereas body motions are mapped to actually command the selected DoF either in EDC or in CC modality.

In the proposed setups, this paradigm is applied to the supernumerary gripper and to the supernumerary arm. In the first case, the user can alternatively control one of the two DoFs of the gripper, each corresponding to the opening/closing of a pair of fingers. In the second case, the user can switch between controlling the gripper complete opening/closing motion or the velocity of the robotic arm along a predefined trajectory.

In both cases, one of the buttons is used to decouple the control of the robot from the user’s motions (i.e., unclutch button). By pressing that button the user can stop the motion of the supernumerary limb and stop controlling it.

In terms of sensory feedback, a different number of vibration bursts delivered through the ring is associated with the press of a certain button. The user also wears the tactile armband that vibrates when the command sent to the supernumerary limb is close to the limits of the range of motion of the SRL. More details on the adopted thresholds and mappings are in Section “Experimental setups”.

Experiments

Experimental setups

Three different robotic systems (supernumerary finger, supernumerary gripper, and supernumerary arm) and three different wearable interfaces (tactile ring, tactile armband, set of wearable IMUs) were adopted. An overview of the technologies is shown in Figure 5.

Figure 5.

Experimental setups. Supernumerary robotic limbs (light blue): (a) finger, (b) gripper, and (c) arm. Wearable interfaces (light orange): (a) tactile ring with buttons, (b) tactile armband and wearable IMUs, and (c) a combination of all the interfaces.

As supernumerary finger, we chose a recently developed prototype of wearable finger presented in Shukla et al. (2024). It features a new module design and tendon routing which allow greater grip stability and better force distribution with respect to previously proposed devices (Salvietti et al., 2016). It has an underactuated tendon-driven structure composed of five rigid phalanges connected through flexible joints, and a single Dynamixel MX-28AT servo motor is used to pull/release the tendon to close/open the finger (Figure 5(a)).

The supernumerary gripper used in the experiments has the same four-fingered soft-rigid tendon-driven structure of the one presented in Salvietti et al. (2018), but has a new compact and robust differential mechanism to actuate the fingers, and an enhanced gripping capability thanks to silicone add-ons on the phalanges. In addition, the rigid ABS phalanges have a more rounded shape for improved safety and likability, and the flexible TPU joints are split in two parts to allow the easy replacement of the tendon and the installation of an extra module if needed. The gripper is underactuated as it features three joints per finger, but has only two servo motors (Dynamixel MX-28AT) that actuate the flexion/extension of each pair of fingers using the differential mechanism. Passive dovetail joints connect the fingers to the palm and allow the user to orient them in the desired direction. The gripper is shown in Figure 5(b) attached to a passive support, and in Figure 5(c) attached to the supernumerary arm. It can either be externally powered or be equipped with an on-board battery.

The adopted supernumerary arm is a Franka Research 3 (Franka Robotics GmbH) collaborative robot equipped with the supernumerary gripper, as shown in Figure 5(c).

The chosen input interfaces are two. The first consists in three push buttons embedded in a ring-shaped interface that also works as a tactile display (Figure 5(a)), the second is composed of four commercial IMUs (Movella DOT, Movella Inc.) of which three are worn on the impaired arm and one is worn on the chest as reference (see Figures 5(b), 5(c)). Data from the IMUs are acquired via Bluetooth using the Movella SDK, which manages possible sensor drift and time synchronization among the sensors. For the ring-shaped interface, pressing any of the three buttons serves as input to the state machine in the EDC modality. In the CC modality, only two of the three buttons are assigned to specific functionalities, while the third button is left unused. In the switching modality, all three buttons are utilized for dedicated functions. Regarding the body tracking interface, in the EDC modality we consider that a trigger is sent to the supernumerary limb when the control signal goes above 0.7.

The adopted feedback interfaces are more compact versions of those previously presented in Torielli et al. (2024) and include an armband and a ring (Figure 6). They both feature an eccentric mass vibration motor for delivering vibrations and a servomotor that allows to wind up a fabric part to deliver skin indentation cues. In this paper, only vibratory stimuli are used. The tactile ring is activated in correspondence to the use of buttons, whereas the tactile armband vibrates when body motions are used. In the EDC modality, one, two, or three vibration bursts are delivered by the tactile interface corresponding to the three states of the supernumerary limb. In the CC modality, a burst composed of three consecutive vibrations alerts the user when the control signal sent to the robot is near to the maximum value (in the experiments the threshold is fixed to 0.9), whereas a single burst of vibration is provided when the value of the control signal is near to its minimum (in the experiments the threshold is fixed to 0.2). In the switching modality, one, two, or three vibration bursts are sent through the tactile ring depending on the pressed button, whereas the tactile armband delivers vibrations according to the above mentioned EDC and CC strategies.

Figure 6.

Wearable tactile interfaces for the finger and the forearm. (a) Armband: (1) PCB with vibromotor, soldering pads for servomotor, and connector for the control board, (2) mechanism for skin indentation, (3) lower case for fixing the interface to the arm, and (4) upper case. (b) Ring: (1) cover for the buttons, (2) PCB with buttons, vibromotor, pads for servomotor, and connector for control board, (3) main structure in ABS plastic, (4) lower part in silicone, and (5) mechanism for skin indentation.

A single control box including a custom-made PCB with an ESP8266 module and a battery controls and powers either both interfaces together or a single one of them. This modular design allows to choose the most suitable combination of interfaces for the task.

The finger, the gripper, and the tactile interfaces are connected to the main computer using ESP compatible dongles. All the adopted supernumerary limbs and wearable sensorimotor interfaces are integrated using the Robot Operating System (ROS 1). During the experiments where body motions were tracked, the control signals derived from real-time computations of relative rotations were transmitted to a ROS environment via User Datagram Protocol (UDP) for further processing.

To facilitate donning and doffing, velcro straps are used for the IMUs, the tactile armband, and the supernumerary finger, whereas the ring features a soft lower part made of silicone which can be changed depending on the size of the user’s finger (Figure 6(b)). Before testing the augmentation systems, participants were helped by experimenters to wear the devices.

The objects used in the experiments are shown in Figure 7 and their properties are summarized in Table 1.

Figure 7.

Objects used in the experiments: (a) with the supernumerary finger and the supernumerary gripper and (b) with the supernumerary arm.

Table 1.

Characteristics of the objects used in the experiments. h and d indicate the height and the maximum diameter of the object, respectively. In item 12, l indicates the length of the pot handle. The toy airplane is from the YCB object set (Calli et al., 2015).

ID	Object	Size (mm)	Weight (g)
1	Rigid glass	h = 98, d = 70	40
2	Soft glass	h = 83, d = 70	3
3	Small jar	h = 63, d = 63	96
4	Big jar	h = 85, d = 74	140
5	Purse	160 × 134 × 50	53
6	Toy airplane	270 × 78 × 78	147
7	Spoon	h = 130, d = 20	46
8	Fork	h = 131, d = 20	41
9	Plastic cube	25 × 25 × 25	4
10	Water bottle	l = 200, d = 50	300
11	Coffee container	h = 146, d = 75	61
12	Pot	h = 85, d = 210,	867
12	Pot	l = 220	867
13	Large spoon	h = 210, d = 30	79
14	Bowl	h = 46, d = 117	34
15	Water container	h = 154, d = 102	121
16	Scale	h = 30, d = 17.5	102

Participants

Two chronic post-stroke patients participated in the conducted study. Their demographic and clinical details are reported in Table 2. The two patients, both right-handed, had similar motor deficits, sparing the proximal segment of the arm and mainly involving the hand ability (they were not able to grasp an object). They had no aphasia. In both cases, the hand paresis followed an ischemic insult of the left middle cerebral artery. Before entering the study, participants were asked to sign an informed consent form. The experiments were approved by the CAREUS Ethical Committee of the University of Siena (Application No. 13/2023). In what follows, we will refer to the participants with P1 and P2.

Table 2.

Participants’ details.

ID	Age	Gender	Injury	Years since injury
P1	63	M	Post-stroke patient with chronic paretic right arm	2
P2	59	M	Post-stroke patient with chronic paretic right arm	12

Experimental protocol

The experimental protocol includes three phases: (i) pre-test phase, (ii) test phase, and (iii) post-test phase.

In the pre-test phase, participants are involved in pre-test interviews to evaluate their life conditions, needs and attitude/expectations toward technology in general and augmentation systems in particular.

In the test phase, after a calibration procedure and some training, patients are involved in experiments with the proposed augmentation systems. After having tested a single experimental condition, they are asked to evaluate their subjective experience through standard questionnaires. The tested experimental conditions and the used evaluation metrics are detailed in Sections “Experimental conditions” and “Evaluation metrics”, respectively. The calibration procedure is performed before the actual test begins, if the tested experimental condition requires the use of body motions. More in detail, the participant, together with the experimenters, first chooses a preferred motion of the impaired limb, and then wears IMU sensors to track it. The participant is asked to repeat the motion, and the collected data are processed as detailed in Section “Extracting a control signal from residual motions”. At the beginning of each experiment, there is a training phase, where each participant is introduced to the specific setup and has 5–10 min to get acquainted with the system. This phase is important to let users familiarize themselves with the control and feedback mappings, including the use of each button when the ring is included in the setup. Once the participant feels confident enough to use the system, the tests begin.

In the post-test phase, at the end of all sessions with a certain supernumerary limb, participants are asked to evaluate their overall experience through an all-encompassing post-test interview. While the pre-test interview is general and has to be done once per participant, the post-test interview is specifically targeted to the used robot.

All the adopted questionnaires are reported in the Supplemental Materials.

Experimental conditions

For each supernumerary limb, different experimental conditions are defined, and each experimental condition is tested in different tasks, which are repeated for a certain number of trials. An overview of the conditions and the tasks is provided in Table 3, the detailed description of the conditions is in Section “Methodology”, whereas the description of the tasks follows.

Table 3.

Tested devices, conditions, and tasks. Each task was performed in all the corresponding conditions for the indicated number of trials. CC: Continuous control; EDC: Event-driven control.

Supernumerary limb	Controlled parameters	Experimental conditions	Tasks	# of trials
Supernumerary finger	1 DoF finger opening/closing	⋅ CC with buttons	⋅ Pour water	6
Supernumerary finger	1 DoF finger opening/closing	⋅ EDC with buttons	⋅ Open/close a jar	6
Supernumerary gripper	1 DoF (4 fingers opening/closing together)	⋅ CC with buttons	⋅ Pour water	6
		⋅ EDC with buttons	⋅ Open/close a purse	2
		⋅ CC with body motions	⋅ Assemble/disassemble a toy	2
		⋅ EDC with body motions	⋅ Assemble/disassemble a toy	2
	2 DoFs (alternative opening/closing of a pair of fingers)	⋅ Buttons to select the pair of fingers, CC with body motions to open/close them	⋅ Pick/place cutlery	6
	2 DoFs (alternative opening/closing of a pair of fingers)		⋅ Place a cube into a glass	6
Supernumerary arm and supernumerary gripper	1 DoF end-effector velocity and	⋅ CC with body motions (buttons for selecting robot)	⋅ Pour water	3
	1 DoF end-effector velocity and	⋅ CC with body motions (buttons for selecting robot)	⋅ Pour coffee	3
	1 DoF gripper closing	⋅ EDC with body motions (buttons for selecting robot)	⋅ Stir content	3
	1 DoF gripper closing	⋅ EDC with body motions (buttons for selecting robot)	⋅ Prepare coffee	1

The tasks are designed to recall everyday life bimanual activities and are performed using objects that are similar to those that anyone could have at home.

The task of pouring water is performed with each of the supernumerary limbs. The supernumerary finger and the supernumerary gripper are used to stabilize a glass that then is filled with water by the participant, whereas the supernumerary arm is used to rotate a bottle and pour water into a container that is held by the user (see Figure 5(c)). When using the supernumerary finger and the supernumerary gripper, two different glasses are manipulated, one made of rigid plastic, and one made of soft, thin plastic. The participant is asked to grasp three times the first and three times the second, in random order. By introducing a fragile, deformable object like the soft plastic glass, we aim at testing the promptness of participants in using the different control modalities, as the participants are asked to delicately grasp the fragile glass and avoid damaging it. With the supernumerary arm, the same pouring motion used for the water is also adopted to pour coffee. To enhance the participants’ engagement and commitment in the tasks with the supernumerary arm, they are asked to pour a certain level of water in a graduated container and a certain weight of coffee in a bowl placed over a scale, without fully emptying the initial containers.

The opening/closing task is performed with both the supernumerary finger and the supernumerary gripper, but with different objects. The supernumerary finger is used to stabilize jars of two different sizes, to test the capability of the participants to unscrew caps that require a certain force (see Figure 5(a)). The order of the jars is randomized, and each jar has to be opened and closed three times. The gripper, instead, is used to open/close a deformable purse that is rather large and needs to be stabilized using two grasping points (see Figure 5(b)). The participant is asked to take the purse from the table, hand it over to the robotic gripper, open it, put an object in it, close it, and then put it back on the table. This operation has to be repeated twice.

Compared to the finger, the gripper provides a more robust grip over the objects, can hold them at a certain height from the table/ground, and has the possibility to actuate two different DoFs. To value these characteristics, additional tasks with it have been devised, as detailed in the following. An assembly task is included among the experiments in which the user can control the closing of all the four fingers together (1 DoF). In particular, the participant has to take a toy plane from the table, hand it over to the gripper, mount an helix on it, place the assembled plane back on the table, hand it over to the gripper again, unmount the helix and lastly place the disassembled plane back onto the table. This procedure has to be performed twice. To exploit the 2 DoFs of the supernumerary gripper, two tasks requiring the gripper to grasp two different objects at the same time are included. The first task consists in picking and placing cutlery. The user is prompted with three forks and three spoons placed on the table on the left side of the gripper and has to move them to the other side of the table and then viceversa. In each trial, the participant has to hand over a fork to a pair of fingers and a spoon to the other pair, and then transport them to the other side of the table moving the passive arm supporting the gripper. This procedure has to be repeated three times from left to right, and three times from right to left. The second task consists in placing a fake ice cube into a glass. The user hands over a glass and a plastic cube to the gripper (glass to the lower pair of fingers, cube to the upper pair), and then releases the cube into the glass (Figure 8). This procedure has to be repeated six times. At the beginning of each trial, the first pair of fingers that has to be used is communicated by the experimenter so that for three times the upper pair is used first, and for three times the lower pair is used first, in random order, for each task.

Figure 8.

Task performed with the supernumerary gripper where the user can control both its DoFs by switching between them using the buttons on the ring interface. The pictures show the actions performed by the user to hand over the objects to the supernumerary gripper. (A), (D) The user activates the intended pair of fingers by pressing the corresponding button on the ring worn on the healthy hand. (B), (E) Then, he proportionally controls the opening/closing of the fingers with the residual motion of the impaired arm. (C), (F) A third button (unclutch button) is used to stop the gripper and decouple its motion from the movement of the participant’s arm.

To show the potentialities of the supernumerary arm in helping with diverse bimanual actions, we also included a task in which the supernumerary arm and the supernumerary gripper are used to stabilize (without lifting) a pot whose content is stirred by the participant. In this case, the participant can switch between controlling the downward/upward motion of the robot arm, and the opening/closing of the gripper. The three tasks performed with the supernumerary arm have been designed to form a sequence of actions that led to the preparation of instant coffee, so after testing the single operations three times with a certain condition, the participant was also asked to perform the entire sequence once.

Evaluation metrics

During the tests, an experimenter records successes and failures of the trials in a predefined scoring sheet. A success is recorded whenever the participant can perform the assigned task in a reasonable amount of time from the beginning to the end, without letting the objects being damaged, slip from the grasp or fall down, and following the task requirements. Failures are recorded whenever the task is not completed. In some tasks, also half-successes are recorded, for example, when users were able to complete the pouring task, but deformed the soft glass too much. Each performed trial is associated with a score of 1 if it is successful, of 0.5 if it is a half-success, or 0 in case of failure. For each task, the scores obtained in all the trials are summed and then divided by the total number of trials to obtain the average success rate that is reported in Section “Results: Task Performance”.

The time required to finish the experimental trials is not considered as an evaluation metric to let the participants perform tasks in a natural way instead of completing them as fast as possible.

After having completed all the tasks of a certain experimental session, in a certain experimental condition, the participant is asked to answer the System Usability Scale (SUS) (Brooke, 1996) and the NASA Task Load Index (NASA-TLX) (Hart, 2006).

The SUS is a well-known questionnaire composed of 10 statements, each of which can be evaluated on a five-point scale that goes from “1 = Strongly Disagree” to “5 = Strongly Agree.” The objective of the SUS is to assess the usability of a certain system, evaluating how much it is easy-to-use and easy-to-learn. The average SUS score for a usable system is considered to be 68 (Sauro, 2011). Obtained results for the SUS are reported and discussed in Section “Usability”.

The NASA-TLX is a widely used, subjective, multidimensional assessment tool for rating perceived workload. Before rating the single factors for the single experimental conditions, participants are asked to perform a pairwise comparison between the sources of workload, indicating which factors contributed most (and least) to the perceived workload during the use of a certain SRL. In this way, each factor is associated with a certain weight, and the overall workload is computed as the weighted average of all the ratings. The participants’ answers are recorded using pen and paper. The answers to the NASA-TLX questionnaire are reported in Section “Perceived Workload”.

Results: Task performance

Supernumerary finger

The results obtained in terms of success rate with the supernumerary finger are reported in Table 4, whereas the signals recorded during a representative trial are shown in Figure 9.

Table 4.

Results obtained with the supernumerary finger.

Conditions	Tasks	P1	P2
CC buttons	⋅ Pour water	100%	92%
CC buttons	⋅ Open/close a jar	67%	100%
EDC buttons	⋅ Pour water	100%	100%
EDC buttons	⋅ Open/close a jar	58%	92%

Figure 9.

Representative trial conducted by P1 with the supernumerary finger using event-driven control with buttons in the pouring task. Each time the user presses a button, the finger sequentially activates the closing motion, stops, and activates the opening, with the user receiving one, two, and three vibration bursts on his finger, respectively.

The supernumerary finger was controlled both in continuous and in event-driven modalities. P1 tested first the event-driven modality and then the continuous one. A total of seven half-successes (4 in CC, 3 in EDC) were recorded when the participant successfully opened the jar using the finger but then closed it without the help of the finger. This likely happened because the paretic hand still has some remaining mobility and can still be used to stabilize objects in tasks that do not require too much force. Only one failure was recorded in EDC, in which the patient could not open the large jar as it slipped from the finger. This happened in the first trial performed with the jars. After that, the participant learned how to stabilize the object well with the supernumerary finger.

P2 tested first the continuous control and then the event-driven one. Only two half successes were observed, one per control condition, as once he squeezed the soft glass too much (without breaking it, anyway), and once he could not properly close the small jar as it was more difficult for him to precisely stabilize it with the supernumerary finger.

Supernumerary gripper

The results obtained in terms of success rate with the supernumerary gripper are reported in Tables 5 and 6 for the two different control strategies, whereas the signals recorded during representative trials are shown in Figures 10 and 11.

Table 5.

Results obtained with the supernumerary gripper when controlling the four-finger opening/closing (1 DoF).

Conditions	Tasks	P1	P2
CC buttons	⋅ Pour	–	100%
	⋅ Open/close	–	100%
	⋅ Assemble	–	100%
EDC buttons	⋅ Pour	–	100%
	⋅ Open/close	–	75%
	⋅ Assemble	–	100%
CC motions	⋅ Pour	100%	100%
	⋅ Open/close	100%	100%
	⋅ Assemble	100%	100%
EDC motions	⋅ Pour	92%	92%
	⋅ Open/close	100%	100%
	⋅ Assemble	100%	100%

Table 6.

Results obtained with the supernumerary gripper when controlling the alternative opening/closing of a pair of fingers (2 DoFs).

Tasks	P1	P2
⋅ Pick and place cutlery	92%	100%
⋅ Place a cube into a glass	100%	100%

Figure 10.

Representative trial conducted with the supernumerary gripper using continuous control with buttons in the task of opening/closing a purse performed by P2 controlling all the four fingers together. While button 2 is pressed, the gripper continuously closes, whereas while button 1 is pressed, the gripper continuously opens. When the command exceeds the upper or lower threshold, three or one vibration bursts are rendered on the user’s finger, respectively.

Figure 11.

Representative trial conducted by P2 with the supernumerary gripper using the 2-DoF control modality. The user exploits the control signal (first row) to control the upper or lower fingers of the gripper (second and third row), depending on the state selected with the buttons (fourth row). The user controls the lower fingers in state 1 and the upper fingers in state 2, whereas in state 3 no fingers are controlled (unclutch). One, two, or three vibrations are rendered on the user’s finger (fifth row) when the state switches to 1, 2, or 3, respectively. When the command exceeds the upper or lower threshold, three or one vibration bursts are rendered on the user’s arm, respectively (sixth row).

Participant P1 completed the first batch of experiments with the supernumerary gripper, that is, those in which the four fingers are closed all together, only using the interface based on body motions, because the interface with buttons was found too difficult to operate. Since the ring must be worn on the healthy hand, it was not easy for him to reach the buttons with the thumb while using the same hand to pass objects to the gripper. This is probably due to the fact that the left hand is not his dominant hand. Anyway, P1 was able to successfully complete almost all trials with the gripper controlled in the two conditions (CC and EDC) exploiting body motions. Only one half-success was scored in the event-driven mode, as the gripper squeezed the soft glass too much and almost crushed it. In the experiment where the user could control separately the upper and lower pair of fingers, a trial was recorded as half success as the grasp of one of the two objects was very loose. However, this did not lead to a failure.

Participant P2 was able to complete all experimental conditions with the supernumerary gripper. During the trials in which the gripper was controlled using buttons, the participant said that he was able to use his healthy hand both to press buttons on the ring interface and to hand over objects to the gripper even if it is not his dominant one. This was possible because after the stroke he learned to use it for every task, and it became very dexterous. This is reflected in the experimental trials, which were all successful except for two half successes that were recorded because once the purse was not completely opened (even though it was open enough to place the object in it) and once the participant almost crushed the soft glass. However, as soon as he noticed this, he immediately opened the gripper a bit, to avoid damaging the object.

Supernumerary arm

The results obtained in terms of success rate with the supernumerary arm are reported in Table 7, whereas the signals recorded during a representative trial are shown in Figure 12. The overall performance of both participants was very successful and only two half successes were observed. In one case, P1 let the bottle of water fall at the end of the pouring task due to poor coordination between pressing the button and catching the bottle with the same hand, whereas at the end of a stirring trial P2 started to raise the supernumerary arm before opening the gripper. However, the pot did not slip and the user could correct his action and finish the task successfully.

Table 7.

Results obtained with the supernumerary arm.

Conditions	Tasks	P1	P2
CC	⋅ Pour water	100%	100%
	⋅ Pour coffee	100%	100%
	⋅ Stir content	100%	83%
	⋅ Full task	100%	100%
EDC	⋅ Pour water	83%	100%
	⋅ Pour coffee	100%	100%
	⋅ Stir content	100%	100%
	⋅ Full task	100%	100%

Figure 12.

Representative trial conducted with the supernumerary arm using event-driven control in the task of pouring water. Each time the control signal exceeds the threshold (first row), it triggers an event on the controlled robot. In the case of the gripper (second row), the first threshold crossing activates the closing motion, the second stops the motion, and the third activates the opening. In the case of the robotic arm (third row), the first threshold crossing moves the robot forward along the pouring trajectory, the second stops the motion, and the third moves the robot backward (the plot shows the pitch of the robot end-effector). The state selected with the buttons (fourth row) determines the controlled robot: state 1 corresponds to the gripper, state 2 to the arm, whereas in state 3 no robots are controlled (unclutch). One or two vibration bursts are rendered on the user’s finger (fifth row) when the state switches to 1 or 2, respectively. One, two or three vibration bursts are rendered on the user’s arm (sixth row) when the signal exceeds the threshold for the first, second or third time for the controlled robot, respectively.

Results: Subjective experience

Usability

Supernumerary finger

The answers to the SUS questionnaire given by P1 considering the use of the supernumerary finger are shown in Figure 13(a). As also confirmed by the post-test interview, P1 overall preferred the continuous control modality with the wearable robotic finger. The event-driven modality was rated as being less easy to learn (U7) and the participant felt less confident when using it (U9). Also, in the event-driven modality the participant gave a neutral answer to U5, which concerns the integration of the system functions. According to him, remembering the use of the buttons was sometimes difficult. Anyway, both control modalities reached a good SUS score (event-driven: 70, continuous: 72.5), above the average value for a usable system (Sauro, 2011).

Figure 13.

Results obtained from the participants’ evaluation of the System Usability Scale (SUS) items considering the use of the supernumerary finger and of the supernumerary gripper: (a), (c) P1; (b), (d) P2.

Regarding the SUS scores given by P2 (Figure 13(b)), overall the usability of the supernumerary finger was rated as rather high (event-driven: 87.5, continuous: 85), above all for the learnability aspect. The participant said that the reason why he rated the event-driven modality with a slightly higher score was because he did not have to remember which button closed the finger and which one opened it.

Supernumerary gripper

The overall SUS scores given by P1 to the tested conditions with the supernumerary gripper are 42.5, 45, and 75 for the event-driven, continuous, and 2 DoFs conditions, respectively. The answers to the single items are reported in Figure 13(c). As it resulted also from the NASA-TLX, the participant appreciated the possibility of controlling 2 degrees of freedom of the gripper. When answering the first item of the usability scale, that is, “U1: I think that I would like to use this system frequently,” for the 2 DoFs condition, the participant said that at the beginning he was skeptical, but after testing the system, he really liked it. He also highlighted that indeed he initially needed guidance to understand how to use the system (in fact, he gave a high rating to U4), but then, after a few minutes of training, he was able to operate the robot by himself, without additional help.

The ratings given by P2 answering the SUS questionnaire relative to the use of the supernumerary gripper with different control modalities are reported in Figure 13(d). Overall, all modalities obtained a good SUS score, that is, 87.5 for the two event-driven control strategies and the 2 DoFs control, and 82.5 for the two continuous control strategies. The latter scored a bit less as according to the participant they required quite some physical effort. Another comment made by the participant was that learning to use the ring interface in the 1-DoF event-driven condition might not be easy for people that are less dexterous than him with the healthy hand.

Supernumerary arm

The supernumerary arm was found to be easy to learn and easy to use by both participants, as the SUS total score given by P1 was equal to 97.5 and 90 for the event-driven and the continuous modalities, respectively, whereas P2 assigned a total SUS score of 95 and 90 to the event-driven and the continuous modalities, respectively. Results are reported in Figure 14. Both participants said they could see themselves using the supernumerary arm frequently in both control modalities, even though they had a preference for the event-driven condition which was deemed less fatiguing and more intuitive.

Figure 14.

Results obtained from the participants’ evaluation of the System Usability Scale (SUS) items considering the use of the supernumerary arm: (a) P1; (b) P2.

Perceived workload

The results related to the NASA TLX are reported according to the guidelines in Hart (2006). Specifically, a graph for each condition shows the scores obtained by the single factors on the vertical axis, while the width of the bars corresponds to the weight given to each factor in the pairwise comparison. Another graph reports the overall workload of all conditions tested for a certain SRL for comparison. The scores of the single factors and the overall workload range between 0 and 100, the weights between 0 and 5.

Supernumerary finger

The scores given by P1 relative to the use of the supernumerary finger are plotted in Figure 15. Due to the reasons explained before, we did not explicitly ask the participants to accomplish tasks as fast as possible, so the Temporal Demand was not perceived as an important factor and scored 0 in the pairwise comparisons of factors. As it can be seen, the Performance factor was deemed very important by the participant, and it obtained a low score in both conditions, which means that P1 felt his performance with the wearable finger was good. Coherently with the answers given in the post-test interview, the continuous control was found to be more natural and thus generated less physical and mental demand on the user, as well as less frustration. This translates in a lower overall workload.

Figure 15.

Results in terms of perceived workload obtained from P1’s ratings of the NASA TLX factors considering the use of the supernumerary finger.

For P2 (see Figure 16), the only two factors contributing to the perceived workload were Mental Demand and Performance, with a reduction in Mental Demand from the continuous control to the event-driven control. This indicates that while he did not perceive that the control modality affected his performance, the event-driven mode required for him less mental effort.

Figure 16.

Results in terms of perceived workload obtained from P2’s ratings of the NASA TLX factors considering the use of the supernumerary finger.

Supernumerary gripper

The results of the NASA-TLX for the supernumerary gripper are shown in Figures 17 and 18 for P1 and P2, respectively. Looking at the overall workload for P1, it can be seen that the conditions in the 1-DoF experiments were rated in a very similar way, whereas the 2-DoF experiment obtained a much lower overall workload, as demonstrated also by the scores assigned to the single factors. The possibility of using 2 DoFs was very appreciated by participant P1, and this probably led to a greater motivation and less perceived effort in the interaction. Another important factor that was underlined by P2, but probably played a role also for P1, is that the presence of the unclutch button, that could be used to stop the gripper and uncouple it from the user, allowed to reduce the physical fatigue during the tasks. Also in the case of the gripper, the Performance factor was considered very important by the participant, and it obtained a low score in all conditions, meaning that P1 perceived his performance as good.

Figure 17.

Results in terms of perceived workload obtained from P1’s ratings of the NASA TLX factors considering the use of the supernumerary gripper.

Figure 18.

Results in terms of perceived workload obtained from P2’s ratings of the NASA TLX factors considering the use of the supernumerary gripper.

In the NASA-TLX, P2 rated the Performance as the most important factor, and he perceived he performed very well in all the experimental conditions (low values of Performance in the graphs indicate a good perceived performance). This is indeed coherent with the fact that he completed all tasks successfully. Other important factors for the participant were the Effort and the Physical Demand which were considered as rather high when using body motions, and in particular in the continuous control condition. As reported by the participant, moving the impaired arm was quite fatiguing, above all in the continuous modality where the user’s arm motion is continuously mapped to that of the robotic gripper. Even though in the 2-DoF experiment the gripper closing motion was controlled in a continuous way as well, the presence of the unclutch button made the task much easier and less tiring for P2. This was underlined by the participant himself in the post-test interview and is reflected in the low values scored by Effort and Physical Demand for the 2 DoFs modality. While the Temporal Demand was not perceived as a very compelling factor, it was rated as rather high in the experiments in which the gripper was controlled using buttons. This is mainly because the user had to do everything with his healthy hand. He had to be quick enough during the handover phase to ensure that the fingers closed over the object and not before placing it between them, and during the release phase to avoid making the object fall down during the opening of the gripper. An interesting thing to notice is that the Frustration factor was deemed as not relevant for this task, as it obtained a weight equal to 0 in the pairwise comparison, and a rating equal to 0 in all conditions. The 2-DoFs condition obtained the lowest overall workload score, as it happened also with P1.

Supernumerary arm

The results of the NASA-TLX for the supernumerary arm are reported in Figures 19 and 20 for P1 and P2, respectively. In this case, the comparison between the two control modalities is more pronounced, as both participants reported a lower workload with event-driven compared to continuous control. This difference can be attributed to the fact that both Effort and Physical Demand for P1, as well as Effort, Physical Demand, Performance, and Temporal Demand for P2, were perceived as slightly present in continuous control, but were experienced as negligible in event-driven control. Regarding Mental Demand, the continuous mode was perceived as slightly more demanding by both participants, though still present for both of them in the event-driven modality. This could be due to the fact that maintaining focus was important to successfully complete all required task steps, regardless of the control modality.

Figure 19.

Results in terms of perceived workload obtained from P1’s ratings of the NASA TLX factors considering the use of the supernumerary arm.

Figure 20.

Results in terms of perceived workload obtained from P2’s ratings of the NASA TLX factors considering the use of the supernumerary arm.

Semi-structured interviews

Pre-test interviews

The pre-test interviews were conducted before the patients were familiarized with the technology of the test. Both patients who were involved in the tests have limited mobility in the right part of their bodies, especially in their arms and hands. Motor functionality and especially grasping or lifting objects is problematic. Both of them are quite autonomous and independent in their everyday lives but still need help from family members. The views on being involved in the study were very positive. While it was the first study participation for one patient, the other one had already been involved in previous studies and had a positive experience. Both expressed a strong sense of respect for scientific research and the potentials that it can have in improving the life situation of stroke survivors.

Post-test interviews

The post-test interviews were conducted shortly after the tests with a specific SRL were completed. The interviews addressed questions related to the patients’ general experience with the control modalities, as well as their thoughts on the potential and limitations of the system.

The wearable finger was perceived as helpful, and understanding how to use it was deemed a quick learning process. The continuous control modality was preferred by P1 as it allowed a higher level of control. A few suggestions for improvements were mentioned, including distancing the buttons on the ring to make them easier to press. Regarding P2, since he tested the early prototypes of the supernumerary finger in previous works, he was able to make a direct comparison with the new version. He acknowledged that it offered a greater strength, which he particularly appreciated when performing tasks such as opening/closing a jar. With the supernumerary finger P2 preferred the event-driven control as it did not require him to remember the exact button to press for closing and for opening.

With respect to the use of the supernumerary gripper, both participants preferred the continuous control modality. For P2, who also tested the ring interface with buttons, the preferred control strategy for the four-finger closing of the supernumerary gripper was the continuous control using the buttons. Unlike what was found with the supernumerary finger, this modality was perceived as easy to use and more precise. The use of body motions was perceived as tiring, so in that case he preferred the event-driven modality. P1 liked the continuous modality better because, even if he found it more tiring compared to the event-driven control, it felt more natural. Both patients appreciated the possibility of controlling two different DoFs of the supernumerary gripper as in this setup the gripper is more versatile and can be used for more complex tasks.

Regarding the supernumerary arm, both participants expressed their preference for the event-driven modality as they found it more intuitive and less tiring. Although their performance was good also in the continuous modality, they felt that in that case they needed to think more about what to do and it was more difficult for them to coordinate the body motions with the use of buttons.

Both patients could envision several use cases in future use of the SRLs, especially in the context of cooking. Vibrotactile acknowledgments were mostly used to reinforce visual feedback to understand when a command was sent to the SRL, above all during the control modalities where participants could switch between different DoFs. When evaluating the SRLs through the post-test questionnaire, both patients expressed that they were satisfied with the dimensions of the devices and stated that they felt safe while using them.

Discussion

Even though the conducted experiments differed in setups and tasks, there are some common advantages and limitations to take away regarding the proposed augmentation systems.

End-users gain effective control over grounded SRLs

The conducted experiments demonstrated that, thanks to wearable sensorimotor interfaces, users can gain an effective control not only on a wearable SRL, but also on SRLs that are physically detached from their bodies. This is shown by the fact that over multiple trials and multiple tasks both participants successfully accomplished the intended operations with minimal training, and always keeping the control over the SRL. Moreover, being the tasks similar to activities of daily living, participants always completed them in a reasonable amount of time. While the missing wearability of the device does not hinder the intuitive control of the robot, it might affect the sense of ownership (Umezawa et al., 2022) and the sense of control (Lisini Baldi et al., 2025), which will be further studied in future works.

End-users can control SRLs using the residual motion of their upper limb

As shown in the graphs reported in Section “Results: Task Performance” and by the successful performance obtained in the experimental trials, both participants were able to modulate the control signal sent to the robot using their body motions. This worked both in continuous and in event-driven control modalities. The extraction of the control signal required a very short calibration process to be performed before starting the trials of a certain experimental session. In the post-test interviews, both participants recognized the importance of keeping the impaired limb active and involved in the task, even though they acknowledged this might cause some fatigue in the long term.

The proposed augmentation systems are easy-to-learn and easy-to-use

With a very short training (few minutes) participants learned how to use the SRLs and succeeded in multiple tasks. They successfully used both an interface based on buttons, and thus similar to remote control devices we use in daily activities, and a more complex system which maps the motion of the impaired limb to the motion of the SRLs. All the usability scores were above the average, except for those related to the control of the supernumerary gripper by one of the participants. Especially the system composed of supernumerary arm and supernumerary gripper was deemed very easy to use, resulting in an average SUS score of SUS_ARM ≈ 93 across both experimental conditions and both participants.

Perceived workload: Low frustration, good performance

The overall perceived workload across all tested augmentation systems and control modalities was relatively low for both participants (on average: OW_P1 ≈ 19, OW_P2 ≈ 23). Frustration was rarely reported, and participants mostly perceived their performance as good. The other factors depended on the adopted control modality.

Vibrotactile feedback useful in specific cases

During the interaction with the SRLs the vibrotactile feedback was found useful in specific cases, such as in the event-driven modality to have an acknowledgment from the robot, in the continuous modality to avoid excessively squeezing the delicate glass, in the stirring task when the supernumerary gripper was occluded by the pot while grasping the handle, and, more in general, to be aware of the selected button. However, even though learning to control the SRLs was fast, still participants’ visual attention was fully dedicated to monitor the task progress and the robot motion. Both participants suggested that probably, to rely more on the tactile feedback and less on the visual one, they would need more acquaintance and familiarity with the system.

Addressing diverse needs of stroke patients

The involved participants were affected in different ways by the stroke and thus have different needs and requirements on assistive devices. In addition, a different amount of time passed since the injury, affecting both the patients’ perception of the impaired arm, and the dexterity of their healthy hand. Therefore, despite the above mentioned common results, still some individual needs and preferences were observed. Future work should consider that stroke can affect individual health and living situation in very different ways (Elnady et al., 2018; Frennert et al., 2024), and thus should focus on strategies to personalize the SRLs control modalities accordingly.

Conclusions and perspectives

This paper proposes a different point of view in the field of human-robot sensorimotor augmentation, introducing grounded supernumerary limbs. The idea is to establish a bilateral connection between a human and a robotic limb not necessarily by coupling them physically, but by letting the user gain control over the SRL through wearable sensorimotor interfaces.

Augmentation systems with different levels of complexity, including a supernumerary finger, a supernumerary gripper and a supernumerary arm, have been designed and tested in different human-robot bimanual tasks with two chronic post-stroke patients. Continuous and event-driven control strategies allowing to command a single DoF of the system have been implemented either using wearable push buttons or by tracking body motions. Combining the two input interfaces, also the possibility of switching between the different DoFs of a single robot or between different robots was included.

The experiments conducted with two end-users revealed the feasibility and usability of the proposed systems, which were evaluated with very positive feedback by participants, who always felt in control of the SRLs and could even imagine themselves using them in their everyday activities. The use of SRLs was deemed easy to learn, and generated a low overall perceived workload.

Users’ preferences in terms of control modality varied across the different experimental setups and conditions. Thus, future work will focus on strategies to learn and implement the personal preferences of the patients, devising, for example, a control system that balances continuous and event-driven modes. Customization options also from the design point of view will be explored to better meet the diverse needs of stroke survivors in terms of comfort and adjustability. For example, since wearing the interfaces and the supernumerary finger currently requires assistance, solutions to facilitate independent donning and doffing will be explored.

Although obtained for a small number of participants, the promising results detailed in this work motivate us to investigate the applicability of the proposed augmentation systems to a larger number of post-stroke patients and to a broader range of end-user categories. In fact, the high learnability, the easiness of use, the good performance and the low frustration level are all factors that candidate SRLs as good assistive tools for all patients with upper limb paresis, including those in whom stroke is not the cause of the deficit (e.g., spinal cord injured patients, multiple sclerosis).

Supplemental Material

Supplemental Material - Wearable and grounded supernumerary robotic limbs for sensorimotor augmentation in post-stroke patients

Supplemental Material for Wearable and grounded supernumerary robotic limbs for sensorimotor augmentation in post-stroke patients by Maria Pozzi, Nicole D’Aurizio, Bernardo Brogi, Giovanni Cortigiani, Leonardo Franco, Manish Shukla, Alessandro Giannotta, Simone Rossi, Sarah Skavron, Susanne Frennert, Gionata Salvietti, Monica Malvezzi, and Domenico Prattichizzo in The International Journal of Robotics Research.

Supplemental Material

Footnotes

ORCID iDs

Maria Pozzi

Nicole D’Aurizio

Giovanni Cortigiani

Funding

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the European Union by the Next Generation EU project ECS17 - PNRR THE (Spoke 9) “THE - Tuscany Health Ecosystem” (PNRR MUR M4 C2 Inv. 1.5, CUP B63C22000680007, Spoke 9: Robotics and Automation for Health) and by the Horizon Europe project “HARIA - Human-Robot Sensorimotor Augmentation - Wearable Sensorimotor Interfaces and Supernumerary Robotic Limbs for Humans with Upper-limb Disabilities” (GA No. 101070292). This work was also supported by the Leonardo S.p.A. under Grant LDO/CTI/P/0025793/22.

Declaration of conflicting interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Supplemental Material

Supplemental material for this article is available online.

References

Ajoudani

Zanchettin

Ivaldi

, et al. (2018) Progress and prospects of the human–robot collaboration. Autonomous Robots 42: 957–975.

Baldi

Spagnoletti

Dragusanu

, et al. (2017) Design of a wearable interface for lightweight robotic arm for people with mobility impairments. In: 2017 International Conference on Rehabilitation Robotics (ICORR). IEEE, 1567–1573.

Ballesteros

Man

Asada

(2023) Supernumerary robotic limbs for next generation space suit technology. In: 2023 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 7519–7525.

Banerjee

Jenamani

Vasudev

, et al. (2024) To ask or not to ask: human-in-the-loop contextual bandits with applications in robot-assisted feeding. arXiv preprint arXiv:2405.06908.

Bhattacharjee

Gordon

Scalise

, et al. (2020) Is more autonomy always better? Exploring preferences of users with mobility impairments in robot-assisted feeding. In: Proceedings of the 2020 ACM/IEEE International Conference on human-robot Interaction, IEEE, 181–190.

Brooke

(1996) Sus: a quick and dirty usability scale. Usability Evaluation in Industry.

Buratti

Deiana

Noccaro

, et al. (2023) Effect of vibrotactile feedback on the control of the interaction force of a supernumerary robotic arm. Machines 11(12): 1085. https://www.mdpi.com/2075-1702/11/12/1085

Calli

Singh

Walsman

, et al. (2015) The ycb object and model set: towards common benchmarks for manipulation research. In: 2015 International Conference on Advanced Robotics (ICAR). IEEE, 510–517.

Ciullo

Veerbeek

Temperli

, et al. (2020) A novel soft robotic supernumerary hand for severely affected stroke patients. IEEE Transactions on Neural Systems and Rehabilitation Engineering: A Publication of the IEEE Engineering in Medicine and Biology Society 28(5): 1168–1177.

10.

Dennler

Cain

De Guzman

, et al. (2023) A metric for characterizing the arm nonuse workspace in poststroke individuals using a robot arm. Science Robotics 8(84): eadf7723.

11.

Dominijanni

Shokur

Salvietti

, et al. (2021) The neural resource allocation problem when enhancing human bodies with extra robotic limbs. Nature Machine Intelligence 3(10): 850–860.

12.

Zhang

Leonori

, et al. (2023) Bi-directional human-robot handover using a novel supernumerary robotic system. In: 2023 IEEE International Conference on Advanced Robotics and Its Social Impacts (ARSO). IEEE, 153–158.

13.

Elnady

Mortenson

Menon

(2018) Perceptions of existing wearable robotic devices for upper extremity and suggestions for their development: findings from therapists and people with stroke. JMIR rehabilitation and assistive technologies 5(1): e9535.

14.

Frennert

Pozzi

Persson

, et al. (2024) Towards a hierarchical user requirement structure for upper limb assistive robotics. ACM Transactions on Human-Robot Interaction 14(1): 1–26.

15.

Goldau

Shastha

Kyrarini

, et al. (2019) Autonomous multi-sensory robotic assistant for a drinking task. In: 2019 IEEE 16th International Conference on Rehabilitation Robotics (ICORR). IEEE, 210–216.

16.

Guggenheim

Asada

(2020) Inherent haptic feedback from supernumerary robotic limbs. IEEE Transactions on Haptics 14(1): 123–131.

17.

Hart

(2006) Nasa-task load index (Nasa-tlx); 20 years later. In: Proceedings of the Human Factors and Ergonomics Society - Annual Meeting. Sage publications Sage CA, Vol. 50, 904–908.

18.

Hasanen

Awad

Boushaki

, et al. (2022) Novel supernumerary robotic limb based on variable stiffness actuators for hemiplegic patients assistance. In: 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 1892–1899.

19.

Hasanen

Suthar

Zweiri

, et al. (2024) Design of twisted string actuated flexure joint for supernumerary robotic arm for bi-manual tasks. IEEE Sensors Journal.

20.

Huo

Wang

Guo

, et al. (2023) Reconfigurable design and modeling of an underwater superlimb for diving assistance. Advanced Intelligent Systems 5(11): 2300245.

21.

Hussain

Meli

Pacchierotti

, et al. (2017) A soft robotic supernumerary finger and a wearable cutaneous finger interface to compensate the missing grasping capabilities in chronic stroke patients. In: 2017 IEEE World Haptics Conference (WHC). IEEE, 183–188.

22.

Inami

(2023) To transformation, duplication, and fusionary—the future of humankind through the jizai body. In: Theory of JIZAI Body: Towards Mastery over the Extended Self. Springer, 1–23.

23.

Jaeger

Baptista

Basla

, et al. (2023) How the cybathlon competition has advanced assistive technologies. Annual Review of Control, Robotics, and Autonomous Systems 6: 447–476.

24.

Jenamani

Stabile

Liu

, et al. (2024) Feel the bite: Robot-assisted inside-mouth bite transfer using robust mouth perception and physical interaction-aware control. In: Proceedings of the 2024 ACM/IEEE International Conference on Human-Robot Interaction, IEEE, 313–322.

25.

Khoramshahi

Poignant

Morel

, et al. (2023) A practical control approach for safe collaborative supernumerary robotic arms. In: 2023 IEEE International Conference on Advanced Robotics and its Social Impacts (ARSO). IEEE, 147–152.

26.

Lee

Eden

Gurgone

, et al. (2024) Control limitations in the null-space of the wrist muscle system. Scientific Reports 14(1): 20634.

27.

Lisini Baldi

D’Aurizio

Gaudeni

, et al. (2024) Exploiting body redundancy to control supernumerary robotic limbs in human augmentation. The International Journal of Robotics Research ■■■: ■■■.

28.

Lisini Baldi

Brogi

Giannotta

, et al. (2025) Quantifying the sense of control through the hand blink reflex in human-robot interaction. IEEE Transactions on Affective Computing ■■■: ■■■.

29.

Liu

Zhu

Zhang

, et al. (2020) Otariidae-inspired soft-robotic supernumerary flippers by fabric Kirigami and origami. IEEE/ASME Transactions on Mechatronics 26(5): 2747–2757.

30.

Luo

Gong

, et al. (2021) Modeling and balance control of supernumerary robotic limb for overhead tasks. IEEE Robotics and Automation Letters 6(2): 4125–4132.

31.

Maheu

Archambault

Frappier

, et al. (2011) Evaluation of the jaco robotic arm: Clinico-economic study for powered wheelchair users with upper-extremity disabilities. In: 2011 IEEE International Conference on Rehabilitation Robotics. IEEE, 1–5.

32.

Nanavati

Ranganeni

Cakmak

(2023) Physically assistive robots: a systematic review of mobile and manipulator robots that physically assist people with disabilities. Annual Review of Control, Robotics, and Autonomous Systems 7: 123–147.

33.

Pinardi

Longo

Formica

, et al. (2023a) Impact of supplementary sensory feedback on the control and embodiment in human movement augmentation. Communications Engineering 2(1): 64.

34.

Pinardi

Noccaro

Raiano

, et al. (2023b) Comparing end-effector position and joint angle feedback for online robotic limb tracking. Plos one 18(6): e0286566.

35.

Poignant

Jarrassé

Morel

(2024) Hands-free teleoperation of a nearby manipulator through a virtual body-to-robot link. arXiv preprint arXiv:2406.09301.

36.

Prattichizzo

Salvietti

Chinello

, et al. (2014) An object-based mapping algorithm to control wearable robotic extra-fingers. In: 2014 IEEE/ASME International Conference on Advanced Intelligent Mechatronics. IEEE, 1563–1568.

37.

Prattichizzo

Pozzi

Baldi

, et al. (2021) Human augmentation by wearable supernumerary robotic limbs: review and perspectives. Progress in Biomedical Engineering 3(4): 042005.

38.

Rand

Eng

(2012) Disparity between functional recovery and daily use of the upper and lower extremities during subacute stroke rehabilitation. Neurorehabilitation and Neural Repair 26(1): 76–84.

39.

Reis

Bernardo

Oshiro

, et al. (2021) Effects of robotic therapy associated with noninvasive brain stimulation on upper-limb rehabilitation after stroke: systematic review and meta-analysis of randomized clinical trials. Neurorehabilitation and Neural Repair 35(3): 256–266.

40.

Saberpour Abadian

Otaran

Schmitz

, et al. (2023) Computational Design of Personalized Wearable Robotic Limbs. UIST ’23. Association for Computing Machinery.

41.

Salvietti

Hussain

Cioncoloni

, et al. (2016) Compensating hand function in chronic stroke patients through the robotic sixth finger. IEEE Transactions on Neural Systems and Rehabilitation Engineering: A Publication of the IEEE Engineering in Medicine and Biology Society 25(2): 142–150.

42.

Salvietti

Iqbal

Hussain

, et al. (2018) The co-gripper: a wireless cooperative gripper for safe human robot interaction. In: 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 4576–4581.

43.

Sauro

(2011) A Practical Guide to the System Usability Scale: Background, Benchmarks & Best Practices. Measuring Usability LLC.

44.

Shukla

Franco

Prattichizzo

, et al. (2024) Enhancing grasping capabilities in stroke patients through a novel mechanical design of the robotic sixthfinger. In: 2024 10th IEEE RAS/EMBS International Conference for Biomedical Robotics and Biomechatronics (BioRob). IEEE, 1752–1757.

45.

Takashita

Arai

Saito

, et al. (2024) Embodied tentacle: mapping design to control of non-analogous body parts with the human body. In: Proceedings of the CHI Conference on Human Factors in Computing Systems, 1–19.

46.

Torielli

Franco

Pozzi

, et al. (2024) Wearable haptics for a marionette-inspired teleoperation of highly redundant robotic systems. In: 2024 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 15670–15676.

47.

Trompetto

Catalano

Farina

, et al. (2022) A soft supernumerary hand for rehabilitation in sub-acute stroke: a pilot study on spasticity.

48.

Fang

Leng

, et al. (2023) Task-based human-robot collaboration control of supernumerary robotic limbs for overhead tasks. IEEE Robotics and Automation Letters 8(8): 4505–4512.

49.

Umezawa

Suzuki

Ganesh

, et al. (2022) Bodily ownership of an independent supernumerary limb: an exploratory study. Scientific Reports 12(1): 2339.

50.

Verdel

Eden

Cervantes-Culebro

, et al. (2024) A predictive coding framework for safe and versatile control of supernumerary robotic limbs.

51.

Zhang

Huang

, et al. (2024) A soft supernumerary robotic limb with fiber-reinforced actuators. Journal of Intelligent and Robotic Systems 110(2): 71.

52.

Zhang

Demiris

(2022) Learning garment manipulation policies toward robot-assisted dressing. Science Robotics 7(65): eabm6010.

53.

Zhang

Balatti

Leonori

, et al. (2023) A human motion compensation framework for a supernumerary robotic arm. In: 2023 IEEE-RAS 22nd International Conference on Humanoid Robots (Humanoids), IEEE, 1–8.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

1.06 MB