A transhumeral prosthesis with an artificial neuromuscular system: Sim2real-guided design,modeling,and control

Abstract

In this work we introduce a new type of human-inspired upper-limb prostheses. The Artificial Neuromuscular Prosthesis (ANP) imitates the human neuromuscular system in the sense of its compliance, backdrivability, natural motion, proprioceptive sensing, and kinesthetics. To realize this challenging goal, we introduce a novel human-inspired and simulation-based development paradigm to design the prosthesis mechatronics in correspondence to the human body. The ANP provides body awareness, contact awareness, and human-like contact response, realized via floating base rigid-body models, disturbance observers, and joint impedance control—concepts known from established state-of-the-art robotics. The ANP mechatronics is characterized by a four degrees of freedom (dof) torque-controlled human-like kinematics, a tendon-driven 2-dof wrist, and spatial orientation sensing at a weight of 1.7 kg (without hand and battery). The paper deals with the rigorous mathematical modeling, control, design and evaluation of this device type along initially defined requirements within a single prototype only. The proposed systemic and grasping capabilities are verified under laboratory conditions by an unimpaired user. Future work will increase the technology readiness level of the next generation device, where human studies with impaired users will be done.

1. Introduction

The principal goal of an arm prosthesis is to render the natural functionality of a lost limb as close as possible with maximum robustness. This goal has been pursued by generations of engineers in passively and actively controlled prostheses. Based on the significant advances in mechatronics and robotics technology over the last 15 years, astonishing high-tech prostheses have been proposed, which brought us a step closer to reaching the human archetype: achievements like human-like size, weight (Bennett et al., 2016; Johannes et al., 2020; Lenzi et al., 2016; Resnik et al., 2014), torque, and kinematics (Johannes et al., 2020) have been successfully shown for specialized robotic upper limb prostheses.

The next step in the development is the realization of more human-like capabilities in upper-limb prostheses. This may be better understood when looking at the human body: the neuromuscular system stands out for its unmatched actuation in terms of high degrees of freedom, high torque, low friction, and backdrivability. It provides numerous unconscious kinesthetic processes such as gravity compensation and impedance adjustment (Burdet et al., 2001; Franklin et al., 2007), giving the human a feeling of body (Franklin et al., 2007; Proske and Gandevia, 2012), and contact awareness (Bays and Wolpert, 2007; Proske and Gandevia, 2012; Wolpert and Flanagan, 2001). For a prosthesis user, all aforementioned natural functionalities are lost in case of an amputation and are to date not available in state-of-the-art prostheses.

With this work we aim for recreating these natural capabilities with the Artifical Neuromuscular Prothesis (ANP). The ANP shall integrate symbiotically into the human body by providing fundamental kinesthetic, motion, contact response and proprioceptive sensing capabilities inspired by human limbs, which are further denoted as body awareness, contact awareness, human-like kinematics and human-like contact response, see Figure 1.

Figure 1.

Capabilities of the Artificial Neuromuscular Prosthesis (ANP): Body awareness: the knowledge of the device base orientation ${}^{\mathrm{W}}{\mathit{A}}_{\mathrm{b}}$ , joint angles q , kinematics ${}^{\mathrm{W}}{\mathit{T}}_j\left(\mathit{q}\right)$ , rigid-body base attachment wrench ${\widehat{\mathcal{F}}}_{\mathrm{b},\mathrm{g}}$ and rigid-body joint torques ${\widehat{\mathit{\tau }}}_{\mathrm{g}}$ which are used for floating base gravity compensation. Contact awareness: the estimation of the external contact wrench ${\widehat{\mathcal{F}}}_{\text{ext}}$ and its effect on the joint torques ${\widehat{\mathit{\tau }}}_{\text{ext}}$ and base wrench ${\widehat{\mathcal{F}}}_{\mathrm{b},\text{ext}}$ . The total joint torques and base wrenches are $\widehat{\mathit{\tau }}={\widehat{\mathit{\tau }}}_{\mathrm{g}}+{\widehat{\mathit{\tau }}}_{\text{ext}}$ and ${\widehat{\mathcal{F}}}_{\mathrm{b}}={\widehat{\mathcal{F}}}_{\mathrm{b},\mathrm{g}}+{\widehat{\mathcal{F}}}_{\mathrm{b},\text{ext}}$ . The base wrench ${\widehat{\mathcal{F}}}_{\mathrm{b}}$ is monitored to enhance user safety. Human-like kinematics is characterized by four joint axes q₁ − q₄ from elbow to wrist. Human-like contact response allows the control of joint torques τ _d and the active adjustment of joint stiffness and damping K _imp and D _imp.

These challenges require a fundamental reconsideration of design tools in prosthetics:

1. A novel design paradigm for the development of human-like prostheses is necessary, which governs how to design technical components in relation to the human body.

2. New technological approaches and engineering tools are needed, as new functionality shall be integrated into the wearable device without making significant compromises on size, weight, torque-density and degrees of freedom.

The novel design paradigm is used for identifying corresponding working principles between the human body and the artificial prosthesis (i.e., in mechanical structures, sensors, actuators, controllers, and device intelligence). Our working hypothesis is that a human-inspired prosthesis, developed along the principles of the human body, also provides human-like behavior. This hypothesis shall be validated in this work based on the feature requirements listed in Figure 1.

The novel technological approach is to utilize mechatronics and controllers from soft and tactile (Albu-Schäffer et al., 2007a; Haddadin et al., 2022; Hirzinger et al., 2002) and humanoid robotics (Englsberger et al., 2014; Hyon et al., 2017), as these provide powerful solutions for complex robotic systems and active compliance control.

The novel engineering tool for the development of the ANP is a complete mathematical model of the robotic device and its controllers, realized in a digital twin simulation. This allows us to forecast the robot’s physical behavior and to optimize its components in a small-size design, despite the novel capabilities and components, which are introduced in the hardware design.

In the following, the state of the in upper-limb prostheses and the background in human motor control is discussed, before outlining the contributions of this work.

1.1. State of the art

1.1.1. Prosthesis design

Different design strategies have been followed in arm prosthetics with the aim of providing the best user experience for the amputee. Usually, the aim is a balance between functionality, weight, and size. Typical requirements for the design of a mechatronic prosthesis include high payload at high Load-to-Weight ratio (Lenzi et al., 2016) and human-like appearance (Ottobock, 2021) (i.e., human-like kinematics, motion, size and texture)—all of which reflect fundamental capabilities of the human body. A key factor for achieving small, lightweight actuators with high output torque is the use of electromechanical actuators combined with high gear ratios (Johannes et al., 2020; Lenzi et al., 2016; Weir et al., 2008). First, let us consider commercial transhumeral prosthetic devices.

1.1.2. Commercial prostheses

Typically, commercial transhumeral prostheses are equipped with an active elbow for flexion-extension (F/E), a single active dof in the wrist for supination/pronation (S/P) (and possibly one additional passive wrist dof), and a modular hand. Examples for commercial devices are: the UtahArm and Wrist Rotator by Motion Control (Motion Control, Inc., Salt Lake City, Utah, USA) (Fillauer, 2021), the Dynamic Arm and Michaelangelo Hand/Wrist by Ottobock (Ottobock SE & Co. KGaA, Duderstadt, Germany) (Ottobock, 2021), the Boston Digital Arm (Liberating Technologies Inc., Hollison, MA, USA) (Liberatingtech, 2021), and the Touch Bionics iLimb Wrist (Ossur hf, Reykjavic, Iceland) (Ossur, 2021).

Devices with flexion-extension (F/E) of the wrist include the KS-Bionic Hand by Kesheng Prostheses (Shanghai Kesheng Prosthetic Technology Co., Shanghai, China) (Keshen, 2021) and the Powered Flexion Wrist by Fillauer Motion Control (Fillauer Europe AB, Sollentuna, Sweden) (Fillauer, 2021). In contrast to the aforementioned systems, for which wrist modules cannot be combined with S/P, the LUKE Arm (Mobious Bionics LLC, Manchester, NH, USA) (Mobius bionics LLC, 2021) has this ability.

1.1.3. Research prostheses “elbow to wrist”

A transhumeral prosthetic system with a 1-dof wrist (S/P) and a 1-dof elbow (F/E) was proposed in (Bennett et al., 2016). The work focuses on a mechanical design solution that provides backdrivable elbow actuation and fits into an anthropomorphic forearm model. A transhumeral prosthetic system with a 2-dof wrist (S/P, F/E) is the Luke Deka arm. The modular prosthesis, also containing a hand, is distributed by the company Mobius Bionics LLC and is available for shoulder, humeral and radial amputation. A prior research variant of that system was proposed by Resnik et al. in (Resnik et al., 2014). Another highly advanced prosthetic system with a 2-dof wrist (S/P, F/E) and hand is the Rehabilitation Institute of Chicago (RIC) arm (Lenzi et al., 2016). The system consists of a 3-dof arm and a 2-dof hand. The focus of the RIC arm lies in providing miniaturized mechatronics for small form factor without sacrificing degrees of freedom and functionality. This was achieved by compact mechatronic design solutions and high gear ratios. Despite the multi-stage gears, high efficiencies were achieved. The modular prosthetic limb (MPL) is the first reported full transhumeral system with a 3-dof wrist with additional R/U deviation. It also includes a 10-dof hand (Bridges et al., 2011; Johannes et al., 2011, 2020). The goal of this project was to develop a cutting-edge prosthetic arm with human-like kinematics, torque and sensory feedback in an anthropomorphic form factor. In fact, the MPL is the first and only reported system apparently utilizing torque sensing and impedance control, although to the best of the author’s knowledge, these capabilities have not yet been demonstrated in public. The MPL joints use multi-stage gears consisting of cycloid and planetary gears in order to obtain high torques, thus are similar to the concept of the RIC arm. The modular wrist consists of a serial RRR kinematics with a human-like form factor, similar to (Lenzi et al., 2016; Weir et al., 2008). The motor for F/E is implemented in the palm of the hand.

Other related areas to our contribution, such as the background in stand-alone prosthesis wrists, prosthetic human machine interfaces, soft and tactile robotic technology may be found by the interested reader in Section A in the Appendix. In the following, we investigate the background in human neuromechanics to better understand our contribution.

1.2. Human neuromechanics

1.2.1. Motor control

The human motor control system is characterized by relatively precise movement independent of both body orientation with respect to gravity (Carvalho et al., 2008) and visual perception (DiZio and Lackner, 2000; Franklin et al., 2007). Humans are also able to perceive and respond to contact forces, regardless of where along the body the contact occurs. They can distinguish contacts by comparing expected and measured signals (Wolpert and Flanagan, 2001). Furthermore, human limbs exhibit accurate compliant behavior during physical interaction with the environment. In fact, the muscle impedance can be modulated—a skill which is used by humans for learning new motor tasks and compensating for uncertainties (Burdet et al., 2001; Franklin et al., 2007).

1.2.2. Neuromechanics

The musculoskeletal system is tendon-driven with very low friction and inertia (Amadio, 2013; Bartz et al., 2019), which corresponds to high-performance backdrivability in mechanical terms. A multi-modal sensory system provides a wide range of information about the external world and the state of the body, even without considering the sense of vision, see Figure 2 (left). Muscle spindles and the Golgi tendon apparatus, which are integrated in the muscle fibers and tendons, respectively, provide sensory feedback on muscle length, speed of stretching, and proprioceptive force. A wide variety of tactile sensors in the skin provide detailed information about contact points, pressure, and texture. The vestibular system senses body orientation relative to gravity.

Figure 2.

The ANP mimics key components of the human neuromuscular system. The figure compares functional technical and biological modules for the human and the ANP.

In the central nervous system (CNS), the cerebellum, the primary motor cortex, and the spinal cord, are mainly responsible for complex motor control and learning, including adaptation to novel dynamics (Milner and Franklin, 2005). It is believed that this is achieved by learning an internal (inverse) dynamics model of the self and of the environment, which may be referred to as a “body image” (Kawato et al., 1987; Proske and Gandevia, 2012), see Figure 2 (left). Thus, the human does not rely on reactive feedback control only, but is able to deliver predictive feed-forward motor commands (Krakauer et al., 1999). Internal models are also used to detect contacts based on proprioceptive sensory information by comparing the expectation from the internal model with the measured signal (Wolpert and Flanagan, 2001). Furthermore, human motor control can be described as a form of impedance control (Burdet et al., 2001; Hogan, 1985). In summary, Figure 3 (left) gives a system-level overview of human neuromechanics that is most relevant for this work. Table 1 depicts a summary of capabilities, provided by the human archetype, and where these may be found in literature. The interested reader may find more details about the human anatomy in Section A in the Appendix. In the following, we present the contribution of this work.

Figure 3.

The technological approach of the ANP is directly inspired by the principles of human motor control. The left image is a modified version from (Sensinger and Dosen, 2020) and was extended accordingly. *In the terminology of automatic control it is rather a compensation.

Table 1.

Comparison of human archetype and corresponding technical solution.

Feature	Human archetype	Technical solution
Body image (body awareness)	(Carvalho et al., 2008; DiZio and Lackner, 2000; Franklin et al., 2007; Kawato et al., 1987; Proske and Gandevia, 2012)	Floating base gravity compensation
Body image (contact awareness)	(Bays and Wolpert, 2007; Proske and Gandevia, 2012; Wolpert and Flanagan, 2001)	Generalized momentum observer and multibody dynamics model
Stiffness/impedance modulation	(Burdet et al., 2001; Franklin et al., 2007; Hogan, 1985)	Impedance control
Backdrivability of muscles	(Amadio, 2013; Bartz et al., 2019)	Torque controlled actuators

1.3. Contribution

1.3.1. Capabilities

In this work, based on our previous work in Kuehn et al. (Kühn and Haddadin, 2017), we introduce a novel prosthesis paradigm that is systematically inspired by the fundamental design and control properties of the human neuromuscular system. Specifically, the design concept was grounded in sensors, actuators and controllers, which have a direct biological correspondence to the human body, see Figure 2.

The body image (Kawato et al., 1987; Proske and Gandevia, 2012) essentially reflects the kinematic and dynamic model of the human. For the ANP, we propose an artificial body image reproducing its key functionalities. ¹ While the human can determine its body orientation ${}^{\mathrm{W}}{\widehat{\mathit{A}}}_{\mathrm{e}}$ via the spinal cord and the limb kinematics ${}^{\mathrm{e}}{\widehat{\mathit{A}}}_{\mathrm{b}}$ , the orientation of the ANP in space can be obtained by the orientation measurement of its Inertial Measurement Unit (IMU) using ${}^{\mathrm{W}}{\widehat{\mathit{A}}}_{\mathrm{I}}$ , see Figure 2. Similar to the human, the device orientation is then used for computing internal models. Figure 3 depicts the principles of human motor control (left) and our corresponding technological mechatronic solution for the ANP with human-inspired sensing, actuation and control methods (right). Consequently, Figure 3 provides a human-inspired development scheme for upper limb prostheses, explaining the use of sensors, actuators and controllers in relation to the human body. The figure also reveals the control schemes of body awareness, contact awareness and human-like contact response from Figure 1, which are defined as follows:

We define body awareness as the ability of the prosthesis

• to calculate its floating base kinematics ${}^{\mathrm{W}}{\widehat{\mathit{A}}}_{\mathrm{e}}\left(\mathit{q}\right)$ for every prosthesis body j based on the measured joint angles q and the device orientation ^W A _b,

• to compute the model-based joint torques $\widehat{\mathit{\tau }}={\widehat{\mathit{\tau }}}_{\mathrm{g}}\left(\mathit{q},{}^{\mathrm{W}}{\mathit{A}}_{\mathrm{b}}\right)$ and base wrenches ${\widehat{\mathcal{F}}}_{\mathrm{b}}={\widehat{\mathcal{F}}}_{\mathrm{b},\mathrm{g}}\left(\mathit{q},{}^{\mathrm{W}}{\mathit{A}}_{\mathrm{b}}\right)$ , affected by gravity, Coriolis and inertia,

• to utilize ${\widehat{\mathit{\tau }}}_{\mathrm{g}}\left(\mathit{q},{}^{\mathrm{W}}{\mathit{A}}_{\mathrm{b}}\right)$ in the joint torque control of the prosthesis ${\mathit{\tau }}_{\mathrm{d}}={\widehat{\mathit{\tau }}}_{\mathrm{g}}\left(\mathit{q},{}^{\mathrm{W}}{\mathit{A}}_{\mathrm{b}}\right)$ to compensate for its rigid-body dynamics, where τ _d is the desired joint torque of the prosthesis joint torque controller.

Consequently, the prosthesis maintains its angular positions q (i.e., $\dot{\mathit{q}}=\mathbf{0}$ ), for arbitrary poses or motions of q and ^W A _b due to the rigid-body compensation. An occurring external wrench ${\widehat{\mathcal{F}}}_{\text{ext}}$ affects a joint acceleration ² $\ddot{\mathit{q}}\ne \mathbf{0}$ . We define contact awareness as the ability of the prosthesis to estimate

• an external wrench ${\widehat{\mathcal{F}}}_{\text{ext}}$ ,

• its effect ${\widehat{\mathit{\tau }}}_{\text{ext}}$ on the estimated joint torques $\widehat{\mathit{\tau }}$ ,

• and its effect ${\widehat{\mathcal{F}}}_{\mathrm{b},\text{ext}}$ on the estimated base wrench ${\widehat{\mathcal{F}}}_{\mathrm{b}}$ , such that it yields $\widehat{\mathit{\tau }}={\widehat{\mathit{\tau }}}_{\mathrm{g}}+{\widehat{\mathit{\tau }}}_{\text{ext}}$ and ${\widehat{\mathcal{F}}}_{\mathrm{b}}={\widehat{\mathcal{F}}}_{\mathrm{b},\mathrm{g}}+{\widehat{\mathcal{F}}}_{\mathrm{b},\text{ext}}$ .

For the case of no contact wrench, ${\widehat{\mathcal{F}}}_{\text{ext}}=\mathbf{0}$ and ${\widehat{\mathit{\tau }}}_{\text{ext}}=\mathbf{0}$ . Consequently, body awareness equals contact awareness if no contact wrench is present. We define human-like contact response as the ability

• • to render linear joint stiffness and damping according to ${\mathit{\tau }}_{\mathrm{d}}={\mathit{K}}_{\text{imp}}\left(\mathit{q}-{\mathit{q}}_{\mathrm{d}}\right)+{\mathit{D}}_{\text{imp}}\left(\dot{\mathit{q}}-{\dot{\mathit{q}}}_{\mathrm{d}}\right)+{\mathit{\tau }}_{\text{ff}}$ , where K _imp and D _imp are stiffness and damping matrices, q _d is a vector of desired joint angles and τ _ff a vector of the rigid-body compensation,

• • to render backdrivable actuation ³ for the case of K _imp = D _imp = 0.

We define human-like kinematics as the ability of the prosthesis to render the human elbow to wrist kinematics in terms of the joints degrees in elbow (F/E) and wrist motion (F/E, R/U, and S/P).

Finally, Table 1 lists the artificial technical solutions of the ANP and compares them to the related biological capabilities in the human body.

1.3.2. Design paradigm

The aforementioned capabilities require accurate numerical multi-body models and advanced mechatronic solutions. In this work, we introduce a sim2real-guided development approach for upper limb prostheses, which enables the design of such a next generation prosthesis, with close to human-like size, weight and human-like capabilities—in just a single hardware prototype iteration. Based on a digital twin simulation of the prosthesis, a stack of simulative tests is performed to validate all desired capabilities from Figures 1 and 2 and to gain information for component choice and dimensioning, which is then used for elaborating the geometries. Once all tests are successful, the device is built and validated in experiments.

1.3.3. Mechatronics

Key capabilities of the ANP mechatronics, see Figure 4, are (i) the torque-controlled robot joints, (ii) the 4-dof kinematics and (iii) an IMU which measures the orientation of the device for internal models in realtime. On this basis, floating base dynamics, extended momentum observation, and joint-level impedance control are combined within our novel Artificial Neuromuscular Controller, see Figure 3 (right). The included wrist provides a tendon-driven mechatronic solution to bring high-performance torque sensing to upper limb prosthetics. ⁴ The wrist includes human-like kinematics due to the coinciding joint axes (i.e., Δl = 0, see Figure 1) and a particularly low inertia of movable parts (m_wr = 28 g, all diagonal tensor entries J_wr < 240 g cm²), ensuring high mechanical transparency. Thus, the full prosthesis is a hybrid tendon-/non-tendon-driven system. These advanced capabilities are provided in a compact, wearable transhumeral prosthesis with a weight of 1.7 kg without hand and battery.

Figure 4.

Mechatronic solution of the ANP with the modular Softhand Pro 2.

1.3.4. Comparison

Table 2 lists the key capabilities of our device and compares them to the state of the art, revealing that floating base gravity compensation, body & contact awareness and contact handling are novel in upper-limb prosthetics to the best of the author’s knowledge. It also reveals that full 4-dof kinematics and torque control are still rare in prosthetics. Typical standard mechanical specifications of the proposed device, such as weight, length, joint torque and speed are compared to state-of-the-art prostheses in Figure 26 in the Appendix.

Table 2.

Feature comparison of transhumeral prostheses from the state of the art.

1.3.5. Summary

In summary, the core contributions of this paper are:

• I) Capabilities: re-creation of the human neuromuscular system for upper-limb prostheses in terms of body awareness, contact awareness, and human-like contact response (according to definition) providing a human-inspired system behavior, or in other words, a joint-torque-level device autonomy.

• II) Mechatronics: the Artificial Neuromuscular Prosthesis ANP with 4-dof human-like kinematics, full joint torque sensing, IMU, small size and a weight of 1.7 kg (without hand and battery), which emulates the human neuromuscular system in terms of

– multi-modal sensing ⁵ and an internal (floating base) model to enable full body and contact awareness,

– human-like motion and contact response behavior via active backdrivability, (adaptable) impedance control and (floating base) gravity compensation.

• III) Design paradigm: the first prosthesis which was developed in a human-inspired and sim2real-guided design paradigm providing a blueprint for the next generation of robotic prostheses.

• In addition, a hybrid tendon/non-tendon-driven joint torque mapping, leading to a hybrid tendon-/non-tendon driven joint impedance controller.

Control and interaction modes—new to the prosthesis world—are enabled through this design: the ANP provides floating base teaching, in combination with the guidance via interaction forces, as an alternative human machine interface. Body and contact awareness may be used to monitor the vulnerable stump prosthesis connection and to introduce a protective control mechanism. Finally, all methods are validated in simulation and experiments.

The remainder of this paper is structured as follows: Section 2 presents the sim2real-guided design process and discusses the requirements for the development. Section 3 proposes the mechatronic solution of the ANP. The modeling and control of the ANP are shown in Section 4 and Section 5. Experimental results can be found in Section 6 (and the associated simulation results in Section F in the Appendix). The paper is discussed in Section 7. The paper is concluded in Section 8.

2. Design process

In this section, we introduce a so called sim2real-guided design process for upper-limb prostheses. By this, we aim to systematize, formalize and simplify the definition, design, simulation/experimental evaluation of upper-limb prostheses. In this context, also related simulation-based design approaches for general mechatronics and robotic systems (Broenink et al., 2010; Kelemenová et al., 2013; Kellner et al., 2015; Mattingly et al., 2012; Qamar et al., 2011; Zhou and Broenik, 2017) may be mentioned, showing that (also incorporating the insights from the state of the art) a gap for a holistic simulation-based development approach in prostheses and even (force-sensitive) robotic systems exists. The key idea of the sim2real-guided design is a time-domain physics simulation of the prosthesis. This simulation aims to reproduce the full physical behavior of the prosthesis in form of a digital twin—similar to a real device but virtually simulated on a computer. As a comparison: in a traditional design, see Figure 5, the prosthetic device is computed and designed along a couple of extreme cases (e.g., maximum load, acceleration, speed and deflection) which are usually derived from simplified models. At this stage, higher-order, dynamic and nonlinear effects are typically neglected and three dimensional problems are simplified to quasi-static two dimensional ones. The benefit of this approach are small and handy algebraic equations which can be easily evaluated (e.g., in a spreadsheet). As a downside, such a traditional design requires a-priori valid assumptions, simplifications and experience. Due to many simplifications, there remains a high risk for systematic errors and thus the need for an increased number of hardware prototypes. Due to these limitations, we instead follow the sim2real-guided design process with a digital twin simulation of the prosthesis, see Figure 5. Rather than solving simple case-dependent algebraic equations, we perform holistic modeling with all known relevant physical effects of the prosthesis and then perform numerical simulations by solving the nonlinear differential equations of the prosthesis including its full control stack via numerical integration. Instead of evaluating extreme cases only, we interact with the device via trajectories, loads, and interaction forces as in reality and then obtain the admissible values based on the set of all simulated time-steps, for example, by taking their maximum. This approach allows for a versatile evaluation of the device and also provides working controllers. Consequently, by using this approach, many physical effects, including their coupling and dynamics, are considered. The process requires less intuition and experience, is more systematic, reduces the chance for systematic errors in prosthesis design, and thus the number of prototypes (cost, effort and time).

Figure 5.

Comparison of traditional versus sim2real-guided mechatronics design to determine admissible loads, explained by the following illustrative and simple example: in the traditional design, a maximum admissible force F_max may be determined by an equation $F_{\max }=m{\ddot{x}}_{\max }$ with m being a mass and ${\ddot{x}}_{\max }$ being a maximum acceleration. In the sim2real-guided design, a more accurate result is obtained by $F_{\max }=\max \left(\left\{F\left(t_1\right),\cdots ,F\left(t_{\infty }\right)\right\}\right)$ based on the output of the numerical simulation. This approach may also be applied to determine, for example, acceleration, speed and deflection. The figure only shows the numerical simulation aspect of the sim2real-guided design in a simplified manner. The full concept with component selection, simulation and experimental evaluation may be found in Figure 6.

Figure 6 shows a detailed scheme of such a sim2real-guided design process. The figure is essentially a flow chart and shows the particular steps of the development process including the flow of information (dotted lines). Based on the initial concept from Figures 1–3, the requirements of the prosthesis (i.e., Functional requirements F, Technical requirements T and Specification S) are derived first. Then, a conceptual approach of the prosthesis is elaborated, which includes the kinematic, transmission, sensor and actuation concept. On this basis, a model and control approach is developed which is implemented in a numerical simulation. Disturbing effects such as friction or uncertainties are not considered in the model but may be easily included into the process.

Figure 6.

Single iteration ANP sim2real-guided design process. For the variables definition, please see Section 4, Section 5 and Table 3.

This time-domain simulation plays a key role in the sim2real-guided design process as it provides information about the feasibility of the control concept and about loads acting on components and structure. More specifically, test scenarios are defined which translate the requirements F, T and S into input trajectories, parameter sets, interactions and test metrics M for simulation and experiment. These test scenarios are essentially a batch of all tests which the prosthesis should fulfill in simulation and reality. The simulation is considered successful if all conditions, referred to as Condition1, are fulfilled for the full stack of test scenarios, see Figure 6. This means, the full Artificial Neuromuscular Controller, see Figure 3, should work together with the plant model in simulation which is verified as follows: impedance control is validated by the residuum $\forall t\in {\mathbb{R}}^{+}:\left|\mathit{q}\left(t\right)-{\mathit{q}}_{\mathrm{d}}\left(t\right)\right|<$ 1×10⁻¹ rad between actual and desired joint angles when following a trajectory. The correct implementation of the prosthesis enhanced momentum observer and the robot model are both validated by comparing estimated and modeled states. This is validated by $\left|{\widehat{\mathit{\tau }}}_{\text{ext}}\left(t=\infty \right)-{\mathit{\tau }}_{\text{ext}}\left(t=\infty \right)\right|<$ 1×10⁻¹ Nm, being the residuum of external torques between observer and plant for the steady state. Further model components, such as body and contact awareness, are validated by $\left|{\widehat{\mathcal{F}}}_{\mathrm{b}}\left(t=\infty \right)-{\mathcal{F}}_{\mathrm{b}}\left(t=\infty \right)\right|<$ 1×10⁻¹ Nm or N for the steady state, where ${\widehat{\mathcal{F}}}_{\mathrm{b}}$ and ${\mathcal{F}}_{\mathrm{b}}$ are modeled and actual prosthesis base wrenches. In addition to that, the acceleration of the joints shall be $\forall t\in {\mathbb{R}}^{+}:\ddot{\mathit{q}}\left(t\right)=\mathbf{0}$ for the gravity compensated system, considering the base to be fixed as t _b = const. The simulated controller should use the same interface, that is, actuator commands and sensor readings, as in the experiments. This is important as the ANP is a hybrid tenon/non-tendon driven system. While all joints utilize a cascaded joint-level impedance controller in combination with an underlying joint-level torque controller, for joint 3 and 4, mathematical mappings are required to construct virtual joint-level torque controllers out of the three tendon force controllers. Non-measurable information shall be calculated mathematically, if possible, that is, via mappings, numerical solvers or observers. For that, actuator coordinates such as angles q _a and torques τ _a shall be transformed to joint angles and torques by { q _a, τ _a} → { q , τ } and vice versa. The use of the exact same controller module for simulations and experiments is an approach for dealing with system complexity (i.e., by early fault elimination, module testing and fast experimental control implementation), and thus an essential method for developing an advanced robotic prosthesis.

In the next step, the mechatronic design is elaborated. This process includes the choice of suitable components (i.e., gears and motors) which provide sufficient performance to realize all simulated values of q _a, τ _a in the smallest possible design. For this, q _a, τ _a are compared to motor and gear data, which are provided by component manufacturers. After a successful choice of components, the prosthesis geometry is designed, validated and manufactured. The detailed mechatronic design decision tree, denoted as Condition2, is explained in Section E in the Appendix.

The manufactured prosthesis is tested experimentally considering the specific Test scenarios, see Figure 6. Finally, the ANP design instance is evaluated by the metrics M from Table 3. The table includes the full requirements F, test scenarios and test metrics M, which are also labeled in the figure captions in the following if fulfilled. Consequently, Table 3 fulfills a key role in the feature evaluation of the ANP as it allows to verify or falsify a feature against specification.

Table 3.

Evaluation of the control features and performance of the ANP as part of Figure 6, If not stated otherwise q _d = 0, D _imp = 0.7 × I , φ _b = (−90,180,90) ^T deg, ${\mathcal{F}}_{\text{ext}}=\mathbf{0}$ and m_l = 0, I_l = 0. The utilized parameters are: continuous arbitrary functions f (t), g (t), h (t); joint damping d ; vector with small values ϵ ; Dirac impulse δ(t); unit step σ(t); actual start, actual final and desired final object translation and orientation are ^W r _ov0, ^W r _ov∞^W r _ov∞d, ^W φ _ov0, ^W φ _ov∞^W φ _ov∞d for attached object v; properties of the attached object are {m_l, I _l,⁴ T _l} with object mass, inertia tensor and attachment frame to the center of mass; joint limitations q _min and q _max, and maximum actuator torques/forces τ _{a, max} = {τ_max,i, ..F_{t, max ,i}} may be found in Table 5. For admissible controller errors q_δ,min, e_imp,max, $e_{\widehat{\tau },\max }$ , e_hum,max, and E, see Table 6. The stiffness values for case 1 and 2, high and low stiffness, are denoted as K _imp,1 and K _imp,2 with belonging joint and torque measurements Δ τ ₁, Δ τ ₂ and Δ q ₁, Δ q ₂ related to a non-deflected steady state. The potential energy of the system is U.

Requirement	Test	Test scenario setup	Test metrics M	Metrics label	Table	Fig. Sim	Fig. Exp
4-dof motion control (F4)	Joint angle tracking with/without load	Kimp ≥ 0	| q − q d| < e imp, max	M 1	-	21	25, 35
		q d = f (t)	| τ − τ d| < eτ, max	M 2
		{ml, I l,4 T com,l}	| F t − F td| < eFt, max	M 3
Human-like motion (F5)	Test human trajectories	Kimp ≥0	| q − q d| < e hum, max	M 4	8	-	-
		q d = f (t), φ b = g (t)	| τ a| < τ a, max	M 5
		Trajectories from Hu et al. (2018), Averta et al. (2020, 2021)	q min < q < q max	M 6
Human-like contact response (F6, F7)	Stiffness variation with contact	0 ≤ Kimp,1 ≤ Kimp,2	Δτ = | τ (t0) − τ (t ∞ )|	M 7	-	21	25
		${\mathcal{F}}_{\text{ext}}=\sigma \left(t-t_0\right){\mathcal{F}}_{\text{ext}}^{\prime }$	Δ q = | q (t0) − q (t ∞ )|
			K imp,1 ≈ Δτ1/Δ q 1 ≤ K imp,2 ≈ Δτ2/Δ q 2
	Contact response with zero stiffness (backdrivability)	Kimp = 0 ∧d = ϵ	${\int }_0^{\infty }{\dot{\mathit{q}}}^T{\widehat{\mathit{\tau }}}_{\text{ext}}\mathrm{d}t<E$	M 8	-	29	12
		${\mathcal{F}}_{\text{ext}}=\delta \left(t-t_0\right){\mathcal{F}}_{\text{ext}}^{\prime }$	Δ q > qδ,min	M 9
Body awareness (F8, F9)	Gravity compensation with contact	Kimp = 0 ∧d = ϵ	$\dot{\mathit{q}}\left(t_{\infty }\right)\approx \mathbf{0}$	M 10	-	29	12
		${\mathcal{F}}_{\text{ext}}=\delta \left(t-t_0\right){\mathcal{F}}_{\text{ext}}^{\prime }$	$\raisebox{1ex}{$\delta $}\!\left/ \!\raisebox{-1ex}{$\delta \mathit{q}$}\right.U(\mathit{q}(t)){|}_{\mathit{q}=\mathit{q}\left(t_{\infty }\right)}\ne \mathbf{0}$	M 11
			${\widehat{\mathit{\tau }}}_{\text{ext}}\left(t_{\infty }\right)<e_{\widehat{\tau },\max }$	M 12
	Gravity compensation with varying orientation Input estimation	Kimp = 0	${\widehat{\mathit{\tau }}}_{\mathrm{g}}\widehat{=}{\mathit{\tau }}_{\mathrm{g}}$ , ${\widehat{\mathcal{F}}}_{\mathrm{b}}\widehat{=}{\mathcal{F}}_{\mathrm{b}}$	M 13	-	30	12
		q = f (t)	$\delta /\delta \mathit{q}U(\mathit{q}(t)){|}_{\mathit{q}=\mathit{q}\left(t_{\infty }\right)}\ne \mathbf{0}$	M 14
		φ b = g (t)	$\dot{\mathit{q}}\left(t_{\infty }\right)\approx \mathbf{0}$	M 15
			${\widehat{\mathit{\tau }}}_{\text{ext}}\left(t_{\infty }\right)<e_{\widehat{\tau },\max }$	M 16
Contact awareness (F10, F11, F12)	Input estimation with contact	Kimp ≥0	${\widehat{\mathit{\tau }}}_{\text{ext}}\widehat{=}{\mathit{\tau }}_{\text{ext}}$ , ${\widehat{\mathcal{F}}}_{\text{ext}}\widehat{=}{\mathcal{F}}_{\text{ext}},{\widehat{\mathcal{F}}}_{\mathrm{b}}\widehat{=}{\mathcal{F}}_{\mathrm{b}}$	M 17	-	22, 23	15, 13
		q = f (t)
		φ b = g (t)	$\left|{\widehat{\mathit{\tau }}}_{\text{ext}}-{\mathit{\tau }}_{\text{ext}}\right|<e_{\widehat{\tau },\max }$	M 18
		${\widehat{\mathcal{F}}}_{\text{ext}}=\mathit{h}\left(t,\mathit{q},{\mathit{\phi }}_{\mathrm{b}}\right)$
Grasping (F13)	Pick and place	Kimp ≥0	{W r o1∞, ..W r o4∞}	M 19	-	-	18
		{W r o10, ..W r o40}	−{W r o1∞d, ..W r o4∞d} < ϵ
		{W r o1∞d, ..W r o4∞d}	{W φ o1∞, ..W φ o4∞}
		{W φ o10, ..W φ o40}	−{W φ o1∞d, ..W φ o4∞d} < ϵ
		{W φ o1∞d, ..W φ o4∞d}	∧ object grasped = TRUE

In conclusion, the proposed sim2real-guided design process combines the power of a numerical physics simulation with a structured rule-based development to realize an upper-limb prosthesis in a single prototype iteration only.

2.1. Requirements

In the following, functional F and technical requirements T as well as specifications S, from Figure 6, are defined. These complex requirements and specifications shall be achieved by the design solution, simulation and experimental results, as described below.

2.1.1. Functional requirements (F) - mechanics

A transhumeral prosthesis with a 3-dof wrist (F/E, R/U, S/P), not including the hand, is to be developed (F₁). In addition, a modular hand shall be attachable at the wrist (F₂). The prosthesis is wearable by both unimpaired subjects and individuals with a transhumeral amputation. For healthy subjects, the device shall be attached at the center of the upper arm (humerus) via 3D-printed components and Velcro fasteners (for testing reasons). For users with a transhumeral amputation, the device should be worn by a stump interface in future versions (F₃).

2.1.2. Functional requirements (F) - control

The device should provide 4-dof motion control from elbow to wrist which shall be realized by an active compliance controller, controlling force/torque and motion (F₄). More specifically, the prosthesis shall be capable of reproducing real everyday human motion, defined by the human data from Hu et al. (2018), Averta et al. (2020, 2021), as closely as possible (F₅), which shall be validated by a physics simulation of the prosthesis considering its maximum capabilities. The joints of the prosthesis should imitate human muscles in the sense of contact response, joint stiffness and backdrivability (F₆). In gravity compensation, guidance of the device solely via interaction forces shall be possible (F₇). The device shall have a body image/body awareness, including the floating base kinematics and robot dynamics of the mechanical system. This should include the awareness of joint torques and base reaction forces/moments at the prosthesis attachment affected by gravity and orientation (F₈). By this approach, the ANP should be able to compensate its own weight during arbitrary operation (F₉). The device is to be equipped with a contact recognition which estimates the magnitude of the wrench distal from the wrist (F₁₀). With this method, the device can quantify the effect of the contact wrench on the prosthesis base and the residual in the form of base forces and moments (F₁₁). In addition, the device shall be equipped with a protective control mechanism to a safe mode (e.g., to gravity compensation) if contacts induce excessive base moments (F₁₂). Reaching, grasping and placing objects of Activities of Daily Life (ADL) shall be possible (F₁₃).

2.1.3. Technical requirements T

These functional requirements are further specified by the following technical requirements. The human-like contact response shall be realized by joint torque controlled robot joints (T₁). These mimic the backdrivability of human muscles if the desired torque is set to zero. Otherwise, they render arbitrary desired torques as provided by an impedance controller or a gravity compensation. For that, the robot joints need to be equipped with a torque sensor to provide a torque measurement to the torque controller (T₂ to realize F₆). Additionally, the prosthesis shall be equipped with floating base gravity compensation, essential for providing the body awareness (T₃ to provide F₈, F₉). Consequently, the prosthesis requires at least one IMU, which must be implemented in the control cycle in order to provide spatial orientation measurement for the floating base gravity compensation (T₄). An active compliance controller shall be realized by joint impedance control (T₅ to provide F₆, F₄). The underlying contact awareness algorithm shall be a generalized momentum observer (De Luca and Mattone, 2003), extended to the floating base model of the prosthesis system, enabling the measurement of external joint torques (T₆ to provide F₁₀, F₁₁, F₁₂). The overall control architecture shall consist of a low-level and high-level control layer, where the former shall be integrated within the prosthesis mechatronics and should run current and torque control with at least 8 kHz. The high-level (impedance and model-based) control system shall run on an external controller at 1 kHz in hard real-time over the EtherCat protocol (T₇). The wrist axes should coincide in one point to provide a human-like kinematics (T₈). Battery and central processing unit should be outside the device as this version of the prosthesis was mainly developed for lab testing (T₉). Technical and functional requirements F, T as well as test metrics M will be labeled in figures and tables when fulfilled. Tables 4–6 show the specifications of the ANP in different domains.

Table 4.

Size and weight specifications S.

		Unit
Total weight (no hand and battery)	1.70	kg
Weight of hand	0.34	kg
Weight of forearm/elbow joint	0.5	kg
Weight of wrist module	0.4	kg
Payload	1.25	kg
Length (elbow-to-wrist)	350	mm
Forearm diameter	63	mm
Active dof (elbow to wrist)	4	-

Table 5.

Performance design specifications S.

i	1	2	3	4	Unit
q max,i	80	120	72	60	deg
q min,i	−100	−120	−72	−60	deg
${\dot{q}}_{\max \hspace{0.17em},i}$	360	360	60	60	deg/s
τ max,i	30	15	3.5	3.5	Nm
F t, max,i	170	170	170	-	N

Table 6.

Admissible error specification S.

	Type	Sim	Exp	Unit
e imp,max	Joint angle	0.1	3	deg
e τ,max	Torque control	-	0.4	Nm
e Ft,max	Force control	-	5	N
$e_{\widehat{\tau },\max }$	Observer	0.01	0.1	Nm
e hum,max	Reference	3	-	deg
q δ,min	Deflection	20	20	deg
E	Contact energy	0.5	0.5	J

In the following, we describe the mechatronic solution of the transhumeral arm prosthesis.

3. Mechatronic solution

The ANP consists of four active degrees of freedom as elbow (1-dof), forearm rotation (1-dof), wrist (2-dof) and additionally a modular 1-dof hand. Figure 7(a) depicts an exploded view of the system. It consists of a robotic Elbow joint and a Forearm joint (of the same size and type). Both are interconnected by a Rigid structure. These are followed by the Wrist module. The ANP was designed to attach a modular hand. Here, the Softhand Pro 2 (Della Santina et al., 2018) is attached to the device. Figure 7(b) provides an overview of the system, its dimensions and coordinate frames. All modules of the prosthesis were custom-made out of Aluminium EN AW 7075. All custom geometries were realized by Computerized Numerical Control (CNC) milling. Specially stressed geometries have been optimized to the defined loads by a finite element analysis (FEA), see Figure 33 in the Appendix.

Figure 7.

System design of the ANP including mechanical design and software structure.

The electronics architecture and software structure are depicted in Figure 7(f). A control personal computer (PC) (x86) runs the high-level control routines on a Ubuntu 16.04 hard realtime system with Matlab/Simulink (MathWorks, MA, USA) at 1 kHz. The control PC, which is the EtherCat Master, communicates with the actuators (EtherCat Slaves) in realtime. Elbow, forearm and wrist run with individual custom-made PCBs and control software. These provide field oriented current control (PI) in a cascade with a P torque controller at 8 kHz. Additionally, IMU measurements are provided by the elbow and forearm electronics over the EtherCat protocol in realtime. Finally, all actuator dof provide motor side joint position and link side joint torque measurements. The prosthesis desired joint motion ${\mathit{q}}_{\mathrm{d}},{\dot{\mathit{q}}}_{\mathrm{d}}$ and stiffness K _i can be set via User Datagram Protocol (UDP) at 100 Hz. This allows control over a smartphone user interface (UI) (for demonstration purpose), surface electromyography (sEMG), electroencephalography (EEG) or other suitable Human Machine Interfaces (HMIs) in the future. More information on the data acquisition and the data analysis may be found in Section D in the Appendix.

All prosthesis modules run at 24 V. Typical current consumption is 0.7 A in idle mode, that is, holding position in impedance control, and up to 1.4 A for the payload experiments shown in this work. In the following, the mechanical design of the ANP wrist and forearm/elbow are elaborated in more detail.

3.1. ANP wrist

Figure 7(c) depicts the mechanical design of the wrist based on the kinematics shown in Figure 9. The advantage of such a design is the lightweight and space-saving joint design at the intersecting joints, as torque, position sensing and actuation are placed remotely. The wrist consists of three point symmetric actuator modules, each driving one of the three tendons, respectively. As the tendons enter the Eyelet, they are guided via two Pulleys at ax2 and ax3 to the Spool, see Figure 7(c). A Harmonic gearing (i_g = 100, τ_g,max = 1.4 Nm, n_g,max = 10000 1/min, no load starting torque τ_max = 3 mNm) drives the Spool (r_s = 3 mm) and amplifies the motor torque of the brushless DC motor (BLDC, τ_m,N = 10 mNm, n_m,max = 10000 1/min) lying on axis ax0. According to Figure 20 in the Appendix, the smallest available gear variant, with the highest gear ratio and the smallest motor, was chosen. Wave generator (WG), circular spline (CS) and flex spline (FP) of the harmonic gearing are used in the configuration driving, fixed, driven.

The tendon force measurement is based on the novel mechanism shown in Figure 8. A linear strain-gauge based Force sensor (F_s,max = 440 N, accuracy of 0.1% of F_s,max) is located between a Bracket and a movable Lever. The tendon force F_t affects the resulting force F _s = l_t/l_sF_t in the force sensor where l_t and l_s are the distances from the bearing axis ax1 to the force F_t and F_s, respectively. The output signal of the Force sensor is amplified and measured by a 16-bit analogue-digital converter. A 14-bit magnetic Position sensor (accuracy ± 1.8°deg) measures the motor angle at the bottom of the BLDC, where low-level Electronics for motor control and strain gauge amplification are also located. The tendons are 0.4 mm thick, made of Dyneema, and provide a break force of 360 N. A maximum tendon force of 170 N was measured.

Figure 8.

Lever mechanism for measuring the tendon force in the wrist module (Requirement T₂).

3.2. ANP forearm and elbow joint

Elbow joint and forearm joint of the prosthesis are realized by the robotic actuator from Figure 7(d). The actuator was custom-made as no device on the market could meet the specifications from Tables 4 and 5. A suitable brushless direct current (BLDC) motor kit (τ_m,N = 0.25 Nm, n_m,max = 6000 1/min) and a harmonic gearing (i_g = 100, τ_g,max = 30 Nm, n_g,max = 6000 1/min) were chosen by following the procedure from Figure 20 in the Appendix. However, the initial combination of the smallest gear and smallest motor did not lead to a feasible mechanical design. Consequently, larger variants were chosen, which still fit within the demanded forearm diameter, see Table 4. As a consequence, the maximum torque of the joint τ_m,Ni_g ≈ 25 Nm became larger ⁶ than the maximum simulated actuator torque max (τ_a(t)) ≈ 8 Nm for lifting the simulated payload of 1.25 kg. The joint includes a custom-made joint torque sensor, see Figure 7(e). It consists of five spokes inspired by the designs in (Lee et al., 2014; Schiele and Hirzinger, 2011). Four strain gauges were placed at the location of maximum positive and negative deflection (red color), caused by compression and bending, on the upper and lower side of each spoke. The strain gauge signals are amplified and measured by a 16-bit analogue-digital converter. The motor position is measured by a 16-bit encoder (accuracy 0.05 deg). Brakes are omitted to save additional weight.

4. Modeling

In this section, we present a complete dynamic model approach for upper limb prostheses. We subsequently use the prosthesis model for plant modeling, advanced body and contact aware algorithms, as well as for feed-forward control. We first present the floating base robotics model, including contact forces and moments acting on the ANP, and then describe how the tendon-driven wrist is integrated into the complete model structure. We model the prosthesis interaction to the residual limb of the human by a 6-dof spring-damper system which provides the reaction forces and moments for our complete plant modeling.

4.1. Floating base dynamics

Essentially, the ANP can be modeled as a serial kinematic structure which is described by its generalized joint coordinates $\mathit{q}\in {\mathbb{R}}^n$ with i = 1..m joints and j = 1..n bodies in our system, with m = 4 and n = 5. Note that the hand is not part of the current mathematical model. However, it may either be included by considering the prosthesis as a tree structure or by assuming the hand module via an interaction wrench. Kinematically, the structure can be described by the modified Denavit–Hartenberg parameters (Khalil and Kleinfinger, 1986) (MDH) with the frames from Figure 7(b). A floating base is considered, which is expressed by the translation vector ${}^{\mathrm{W}}{\mathit{t}}_{\mathrm{b}}\in {\mathbb{R}}^{3\times 1}$ and the rotational vector ${\mathit{\phi }}_{\mathrm{b}}\in {\mathbb{R}}^{3\times 1}$ . Orientations in this work are defined in roll pitch yaw (RPY) convention, utilizing an orientation vector $\mathit{\phi }\in {\mathbb{R}}^3$ . Thus, the floating base coordinate is

${\mathit{x}}_{\mathrm{b}}={\left(\begin{array}{ll}{}^{\mathrm{W}}{\mathit{t}}_{\mathrm{b}}^T& {\mathit{\phi }}_{\mathrm{b}}^T\end{array}\right)}^T.$

(1)

The matrix

${}^{\mathrm{W}}{\mathit{T}}_{\mathrm{b}}=\left(\begin{array}{ll}{}^{\mathrm{W}}{\mathit{A}}_{\mathrm{b}}\left({\mathit{\phi }}_{\mathrm{b}}\right)& {\mathit{t}}_{\mathrm{b}}\\ \mathbf{0}& 1\end{array}\right)\in \text{SO}\left(3\right)$

(2)

allows mapping from base to world coordinates. The vector

${\mathit{q}}_{\mathrm{c}}={\left(\begin{array}{ll}{\mathit{x}}_{\mathrm{b}}^T& {\mathit{q}}^T\end{array}\right)}^T$

(3)

denotes the generalized coordinates of the system with x _b being the concatenation of translation and rotational base coordinates.

The dynamic floating base model of the prosthesis is

$\mathit{M}\left({\mathit{q}}_{\mathrm{c}}\right){\ddot{\mathit{q}}}_{\mathrm{c}}+\mathit{c}\left({\mathit{q}}_{\mathrm{c}},{\dot{\mathit{q}}}_{\mathrm{c}}\right)+\mathit{g}\left({\mathit{q}}_{\mathrm{c}}\right)=\underset{{\mathit{\tau }}_{\mathrm{c}}}{\underbrace{\left(\begin{array}{l}{\mathbf{0}}_6\\ \mathit{\tau }\end{array}\right)}}+{\mathit{\tau }}_{\text{ext},\mathrm{c}}.$

(4)

$\mathit{M}\left({\mathit{q}}_{\mathrm{c}}\right)\in {\mathbb{R}}^{\left(m+6\right)\times \left(m+6\right)}$ denotes the mass matrix, $\mathit{c}\left({\mathit{q}}_{\mathrm{c}},{\dot{\mathit{q}}}_{\mathrm{c}}\right)\in {\mathbb{R}}^{m+6}$ the Coriolis and centrifugal vector, $\mathit{g}\left({\mathit{q}}_{\mathrm{c}}\right)\in {\mathbb{R}}^{m+6}$ the vector of gravitational forces, $\mathit{\tau }\in {\mathbb{R}}^m$ the vector of joint torques and ${\mathit{\tau }}_{\text{ext},\mathrm{c}}\in {\mathbb{R}}^{m+6}$ the vector of generalized external forces. These can be further subdivided into

$\mathit{M}\left({\mathit{q}}_{\mathrm{c}}\right)=\left(\begin{array}{ll}{\mathit{M}}_{\mathrm{b},\mathrm{b}}\left({\mathit{q}}_{\mathrm{c}}\right)& {\mathit{M}}_{\mathrm{b},\mathrm{j}}\left({\mathit{q}}_{\mathrm{c}}\right)\\ {\mathit{M}}_{\mathrm{j},\mathrm{b}}\left({\mathit{q}}_{\mathrm{c}}\right)& {\mathit{M}}_{\mathrm{j},\mathrm{j}}\left({\mathit{q}}_{\mathrm{c}}\right)\end{array}\right),$

(5)

$\mathit{c}\left({\mathit{q}}_{\mathrm{c}},{\dot{\mathit{q}}}_{\mathrm{c}}\right)=\left(\begin{array}{l}{\mathit{c}}_{\mathrm{b}}\left({\mathit{q}}_{\mathrm{c}},{\dot{\mathit{q}}}_{\mathrm{c}}\right)\\ {\mathit{c}}_{\mathrm{j}}\left({\mathit{q}}_{\mathrm{c}},{\dot{\mathit{q}}}_{\mathrm{c}}\right)\end{array}\right),\mathit{g}\left({\mathit{q}}_{\mathrm{c}}\right)=\left(\begin{array}{l}{\mathit{g}}_{\mathrm{b}}\left({\mathit{q}}_{\mathrm{c}}\right)\\ {\mathit{g}}_{\mathrm{j}}\left({\mathit{q}}_{\mathrm{c}}\right)\end{array}\right)$

consisting of corresponding block matrices with indices b and j and denote a base on base, joint on joint, joint on base and a base on joint effect on the general coordinates q _c, respectively. The vector of overall generalized external forces is

${\mathit{\tau }}_{\text{ext},\mathrm{c}}={\left(\begin{array}{lll}{}^{\mathrm{W}}{\mathit{F}}_{\mathrm{b}}^T& {}^{\phi }{\mathit{M}}_{\mathrm{b}}^T& {\mathit{\tau }}_{\text{ext}}^T\end{array}\right)}^T,$

(6)

with ${}^{\mathrm{W}}{\mathit{F}}_{\mathrm{b}}\in {\mathbb{R}}^3$ being the external forces acting on the base, ${}^{\phi }{\mathit{M}}_{\mathrm{b}}\in {\mathbb{R}}^3$ being the external moments acting on the base in the twisted frame (φ) and ${\mathit{\tau }}_{\text{ext}}\in {\mathbb{R}}^{n\times 1}$ being the external torque acting on the joints q of the serial kinematics. The transformation between moments in world coordinates and twisted coordinates is

${}^{\phi }{\mathit{M}}_{\mathrm{b}}={\mathit{J}}_{\omega }^T{}^{\mathrm{W}}{\mathit{M}}_{\mathrm{b}},$

(7)

with the twist matrix J _ω , which is defined in (45) in the Appendix.

Two types of contacts are considered for the prosthesis summing up to the generalized total forces

${\mathit{\tau }}_{\text{ext},\mathrm{c}}={\mathit{\tau }}_{\text{ext},\mathrm{C}}+{\mathit{\tau }}_{\text{ext},\mathrm{b}},$

(8)

with τ _ext,C being the generalized forces for external contact and τ _ext,b being the generalized forces for base attachment. Contact of the floating base system is modeled by

${\mathit{\tau }}_{\text{ext},\mathrm{C}}={\mathit{J}\left(\mathrm{C},{\mathit{q}}_{\mathrm{c}}\right)}^T{}^{\mathrm{W}}{\mathcal{F}}_{\text{ext}},$

(9)

with $\mathit{J}\left(\mathrm{C},{\mathit{q}}_{\mathrm{c}}\right)\in {\mathbb{R}}^{6\times \left(n+6\right)}$ being the floating base Jacobian of a contact frame ${}^{\mathrm{W}}{\mathit{T}}_{\mathrm{C}}\left({\mathit{q}}_{\mathrm{c}}\right)\in \text{SO}\left(3\right)$ . The derivation of the floating base Jacobian can be found in the Appendix Section B. ${}^{\mathrm{W}}{\mathcal{F}}_{\text{ext}}\in {\mathbb{R}}^6$ is the associated wrench

${}^{\mathrm{W}}{\mathcal{F}}_{\text{ext}}={\left(\begin{array}{ll}{}^{\mathrm{W}}{\mathit{F}}_{\text{ext}}^T& {}^{\mathrm{W}}{\mathit{M}}_{\text{ext}}^T\end{array}\right)}^T,$

(10)

expressed in world coordinates (W) consisting of a force ${}^{\mathrm{W}}{\mathit{F}}_{\text{ext}}\in {\mathbb{R}}^3$ and a moment ${}^{\mathrm{W}}{\mathit{M}}_{\text{ext}}\in {\mathbb{R}}^3$ .

The fixation of the floating base prosthesis to the human is modeled as a spring-damper system. The location of the floating base coordinate frame (b) is assumed to correspond to the attachment point between human residual limb and prosthesis, see Figure 7(b). Let us assume the position and orientation of the human residual limb to be expressed by

${\mathit{x}}_{\mathrm{h}}={\left(\begin{array}{ll}{}^{\mathrm{W}}{\mathit{t}}_{\mathrm{h}}^T& {}^{\mathrm{W}}{\mathit{\phi }}_{\mathrm{h}}^T\end{array}\right)}^T,$

(11)

where ${}^{\mathrm{W}}{\mathit{t}}_{\mathrm{h}}\in {\mathbb{R}}^3$ and ${}^{\mathrm{W}}{\mathit{\phi }}_{\mathrm{h}}\in {\mathbb{R}}^3$ are its translational and rotational components. The attachment of the prosthesis to an arbitrary location or the human limb may be modeled dynamically by the 6-dof-spring/damper-system

${\mathcal{F}}_{\mathrm{b}}={\mathit{K}}_{\mathrm{f}}{\stackrel{\sim }{\mathit{x}}}_{\mathrm{b}}-{\mathit{D}}_{\mathrm{f}}{\dot{\stackrel{\sim }{\mathit{x}}}}_{\mathrm{b}},$

(12)

with ${\stackrel{\sim }{\mathit{x}}}_{\mathrm{b}}={\mathit{x}}_{\mathrm{b}}-{\mathit{x}}_{\mathrm{h}}$ expressing the spring-damper deflection. ⁷ ${\mathit{K}}_{\mathrm{f}}\in {\mathbb{R}}^{6\times 6}$ and ${\mathit{D}}_{\mathrm{f}}\in {\mathbb{R}}^{6\times 6}$ are stiffness and damping matrices for rotation and translation. The estimation of ${\widehat{\mathcal{F}}}_{\mathrm{b}}$ , used in the controller, can be found in Section 5. The effect on the prosthesis base in generalized coordinates can then be described by

${\mathit{\tau }}_{\text{ext},\mathrm{b}}={\mathit{J}\left(\mathrm{b},{\mathit{x}}_{\mathrm{b}}\right)}^T{}^{\mathrm{W}}{\mathcal{F}}_{\mathrm{b}},$

(13)

with $\mathit{J}\left(\mathrm{b},{\mathit{x}}_{\mathrm{b}}\right)\in {\mathbb{R}}^{6\times 10}$ denoting the floating base Jacobian of the base, based on (2).

4.2. Tendon actuation

Even though the prosthesis is finally controlled in joint space, the wrist is locally controlled in tendon space via tendon-level force controllers. This requires back and forth transformations from joint to tendon space and vice versa, see Figure 9(a). As the wrist consists of the last two rotational joints q₃ and q₄ of the structure, see Figure 7(b), the wrist coordinates ${\mathit{q}}_{\mathrm{w}}={\left(q_3,q_4\right)}^T$ and ${\mathit{\tau }}_{\mathrm{w}}={\left({\tau }_3,{\tau }_4\right)}^T$ are introduced.

Figure 9.

Tendon modeling and control. (a) Tendon-joint mappings for the 3SPS-1RR wrist. (b) 3SPS-1RR kinematics of the wrist.

The ANP wrist of the prosthesis can be classified as 3SPS-1RR tendon driven parallel kinematics with S, P, R representing spherical, translation and rotational degrees of freedom. The preceding number (3) denotes the number of serial chains, consisting of a spherical, prismatic and spherical degree of freedom (SPS). These are in parallel to two rotational joints in series (1RR), see Figure 9.

The system is driven by k = 1..3 tendons which are attached at T _k to the endeffector and pass the eyelets B _k . Both T _k and B _k are equally distributed on circles with radius r and R, respectively. T₁ and B₁ lie on the same plane spanned by y and z of frame ^W T _wb and for q _w = 0.

Despite the parallel kinematics of the wrist, the prosthesis can still be expressed as serial chain kinematics as if the tendons are considered massless. However, nonlinear mappings for kinematics and static force/torque relations are required. Following (Murray et al., 1994), the extension function, or inverse kinematics of the wrist

$\begin{array}{l}{\mathit{h}}_1={\mathit{r}}_{\mathrm{B}1\mathrm{w}}+{\mathit{r}}_{\text{wE}}+{\mathit{r}}_{\text{ET}1}\left({\mathit{q}}_{\mathrm{w}}\right)\end{array}$

(14)

$\begin{array}{l}{\mathit{h}}_2={\mathit{r}}_{\mathrm{B}2\mathrm{w}}+{\mathit{r}}_{\text{wE}}+{\mathit{r}}_{\text{ET}2}\left({\mathit{q}}_{\mathrm{w}}\right)\end{array}$

(15)

$\begin{array}{l}{\mathit{h}}_3={\mathit{r}}_{\mathrm{B}3\mathrm{w}}+{\mathit{r}}_{\text{wE}}+{\mathit{r}}_{\text{ET}3}\left({\mathit{q}}_{\mathrm{w}}\right)\end{array}$

(16)

$\begin{array}{l}\mathit{h}\left({\mathit{q}}_{\mathrm{w}}\right)=\left(\begin{array}{l}\sqrt{{h_{1,\mathrm{x}}}^2+{h_{1,\mathrm{y}}}^2+{h_{1,\mathrm{z}}}^2}\\ \sqrt{{h_{2,\mathrm{x}}}^2+{h_{2,\mathrm{y}}}^2+{h_{2,\mathrm{z}}}^2}\\ \sqrt{{h_{3,\mathrm{x}}}^2+{h_{3,\mathrm{y}}}^2+{h_{3,\mathrm{z}}}^2}\end{array}\right)\end{array}$

(17)

is obtained by describing the tendon length h ( q _w) from B _k to T _k for each tendon applying the Euclidean norm. Here, r _Bkw is the vector from point B _k to wrist base (wb), r _wE is the vector from the wrist base to the endeffector E, h_w is a height and r _ETk is the vector from the endeffector to the tendon fixation point T _k . The tendon Jacobian matrix is

$\mathit{P}\left({\mathit{q}}_{\mathrm{w}}\right)={\frac{\partial \mathit{h}\left({\mathit{q}}_{\mathrm{w}}\right)}{\partial {\mathit{q}}_{\mathrm{w}}}}^T\in {\mathbb{R}}^{2\times 3}.$

(18)

The relation between wrist joint torque τ _w and tendon force F _t is

${\mathit{\tau }}_{\mathrm{w}}=\mathit{P}\left({\mathit{q}}_{\mathrm{w}}\right){\mathit{F}}_{\mathrm{t}}.$

(19)

The relation between wrist joint velocity ${\stackrel{.}{\mathit{q}}}_{\mathrm{w}}$ and tendon velocity $\stackrel{.}{\mathit{h}}$ is

$\dot{\mathit{h}}={\mathit{P}}^T\left({\mathit{q}}_{\mathrm{w}}\right){\dot{\mathit{q}}}_{\mathrm{w}}.$

(20)

In the following, the tendon length $\mathit{l}\in {\mathbb{R}}^{3\times 1}$ is utilized which corresponds to the measured value of the extension function $\mathit{h}\left({\mathit{q}}_{\mathrm{w}}\right)\in {\mathbb{R}}^{3\times 1}$ .

The direct kinematics can be computed at each control cycle based on a given tendon length l via the optimization problem (Chalon et al., 2011; Toedtheide et al., 2021)

$\underset{{\mathit{q}}_{\mathrm{w}}}{\text{minimize}} \left(\widehat{\mathit{h}}\left({\mathit{q}}_{\mathrm{w}}\right)-\mathit{l}\right)$

(21)

in realtime, where $\widehat{\mathit{h}}$ is the modeled tendon length. ⁸ Joint velocities are obtained via

${\dot{\mathit{q}}}_{\mathrm{w}}={{\mathit{P}}^T}^{\#}\left({\mathit{q}}_{\mathrm{w}}\right)\dot{\mathit{l}},$

(22)

where ${{\mathit{P}}^T}^{\#}\left(\mathit{q}\right)$ is the pseudo inverse of P ^T ( q ).

The desired tendon forces F _t,d, related to the wrist motor torques, are calculated via linear programming, solving the Karush–Kuhn–Tucker problem (Wu, 2007) by

$\begin{array}{l}\underset{{\mathit{F}}_{\mathrm{t},\mathrm{d}}}{\min } \left(\begin{array}{lll}1& 1& 1\end{array}\right){\mathit{F}}_{\mathrm{t},\mathrm{d}}\\ \text{such} \text{that}\\ {\mathit{F}}_{\mathrm{t},\mathrm{d}}\ge {\mathit{F}}_{\mathrm{t},\mathrm{p}}\\ \mathit{P}\left({\mathit{q}}_{\mathrm{w}}\right){\mathit{F}}_{\mathrm{t},\mathrm{d}}={\mathit{\tau }}_{\mathrm{w},\mathrm{d}}\end{array}$

(23)

in realtime with F _t,p being a minimum desired pretension force vector and ${\mathit{\tau }}_{\mathrm{w},\mathrm{d}}=\left({\mathbf{0}}_{2\times 2},{\mathit{I}}_{2\times 2}\right){\mathit{\tau }}_{\mathrm{d}}$ .

By defining the actuator coordinates of the whole prosthesis to

$\begin{array}{l}{\mathit{q}}_{\mathrm{a}}={\left(\begin{array}{lllll}{q}_1& q_2& l_1& l_2& l_3\end{array}\right)}^T,\end{array}$

(24)

$\begin{array}{l}{\mathit{\tau }}_{\mathrm{a}}={\left(\begin{array}{lllll}{\tau }_1& {\tau }_2& \frac{F_{\mathrm{t}1}}{r_{\mathrm{s}}}& \frac{F_{\mathrm{t}2}}{r_{\mathrm{s}}}& \frac{F_{\mathrm{t}3}}{r_{\mathrm{s}}}\end{array}\right)}^T,\end{array}$

(25)

with r_s being the spool radius, the tendon module can be further abstracted. This is achieved by introducing identity matrices for the rotary joints 1 and 2 and by placing the outcome of the respective tendon model on subsequent matrix diagonal elements. These operations can be found in Section C in the Appendix utilizing (17)–(23). As the consequence, a mapping from tendon-level to joint-level, and vice versa, for the whole hybridly actuated prosthesis is obtained. This enables the control of the system in joint coordinates q and τ , without the need to consider the actuator level coordinates.

5. Control

The two main control methods in the prosthesis are a fully fledged joint impedance controller and a floating base generalized momentum observer, that is, well established methods from soft and tactile robots, yet not established in upper limb prosthetics.

5.1. Joint impedance control

The prosthesis is controlled by the standard joint impedance controller

${\mathit{\tau }}_{\mathrm{d}}={\mathit{K}}_{\text{imp}}\left(\mathit{q}-{\mathit{q}}_{\mathrm{d}}\right)+{\mathit{D}}_{\text{imp}}\left(\dot{\mathit{q}}-{\dot{\mathit{q}}}_{\mathrm{d}}\right)+{\mathit{\tau }}_{\text{ff}}+{\mathit{\tau }}_{\text{vw}},$

(26)

with K _imp and D _imp being the desired stiffness and damping matrices and q _d desired joint angles, which may also be obtained by velocity inputs ${\mathit{q}}_{\mathrm{d}}={\int }_0^{\tau }{\stackrel{.}{\mathit{q}}}_{\mathrm{d}}dt$ . A feed forward floating base Coriolis and gravity model compensation

${\mathit{\tau }}_{\text{ff}}={\widehat{\mathit{c}}}_{\mathrm{j}}\left({\mathit{q}}_{\mathrm{c}},{\dot{\mathit{q}}}_{\mathrm{c}}\right){\dot{\mathit{q}}}_{\mathrm{c}}+{\widehat{\mathit{g}}}_{\mathrm{j}}\left({\mathit{q}}_{\mathrm{c}}\right)$

(27)

utilizes the suitable block matrices from (5). This differs from the regular fixed base model by considering both the joint and the base orientation measurements q and φ _b in the model. Virtual joint limits τ _vw are realized by the joint-wise case distinction

${\tau }_{\text{vw}i}=\{\begin{array}{ll}0& \text{if} q_i>q_{\min \hspace{0.17em},i}\wedge q_i<q_{\max \hspace{0.17em},i}\\ K_{\mathrm{w},i}\left(-q_i+q_{\min }\right)& \text{if} q_i\le q_{\min \hspace{0.17em},i}\\ K_{\mathrm{w},i}\left(q_i-q_{\max }\right)& \text{if} q_i\ge q_{\max \hspace{0.17em},i},\end{array}$

(28)

with K_w,i, q_min,i and q_max,i being the wall stiffness, minimum and maximum joint position for joint i. The well-known diagonal damping approach (Albu-Schäffer and Hirzinger, 2003) is applied to

${\mathit{D}}_{\text{imp}}=\sqrt{\widehat{\mathit{M}}\left(\mathit{q}\right){\mathit{D}}_{\mathrm{\Psi }}}\sqrt{{\mathit{K}}_{\text{imp}}}+\sqrt{{\mathit{K}}_{\text{imp}}\left(\mathit{q}\right){\mathit{D}}_{\mathrm{\Psi }}}\sqrt{\widehat{\mathit{M}}\left(\mathit{q}\right)},$

(29)

with D _Ψ being the unified damping matrix.

5.2. Momentum observation

The generalized floating base momentum observer

${\widehat{\mathit{\tau }}}_{\text{ext}}={\mathit{K}}_{\mathrm{O}}\left({\int }_0^T\left(\mathit{\tau }-\mathit{\beta }\left({\mathit{q}}_{\mathrm{c}},{\dot{\mathit{q}}}_{\mathrm{c}}\right)-{\widehat{\mathit{\tau }}}_{\text{ext}}\right)\text{dt}-{\widehat{\mathit{M}}}_{\mathrm{j},\mathrm{j}}\left({\mathit{q}}_{\mathrm{c}}\right)\dot{\mathit{q}}\right)$

(30)

from (De Luca and Mattone, 2003; Haddadin et al., 2017), is used for estimating joint torque, where

$\mathit{\beta }\left({\mathit{q}}_{\mathrm{c}},{\dot{\mathit{q}}}_{\mathrm{c}}\right)={\widehat{\mathit{c}}}_{\mathrm{j}}\left({\mathit{q}}_{\mathrm{c}},{\dot{\mathit{q}}}_{\mathrm{c}}\right)+{\widehat{\mathit{g}}}_{\mathrm{j}}\left({\mathit{q}}_{\mathrm{c}}\right)-{\dot{\widehat{\mathit{M}}}}_{\mathrm{j},\mathrm{j}}\left({\mathit{q}}_{\mathrm{c}}\right)\dot{\mathit{q}}$

(31)

for our floating base system. The joint torque uses $\mathit{\tau }={\left({\tau }_1,{\tau }_2,{\tau }_{\text{wr}1},{\tau }_{\text{wr}2}\right)}^T$ with τ_wr1, τ_wr2 from the virtual joint torque sensor of the wrist (19) as proposed in (Toedtheide et al., 2021). As a reference, the ability to estimate contacts by proprioceptive measurements can also be found in human biology (Proske and Gandevia, 2012).

With this basis, the algorithms for body and contact awareness can be built in the following.

5.3. Body and contact awareness

In the following, a model is derived which is then used for computing the contact wrench ${}^{\mathrm{W}}{\mathcal{F}}_{\mathrm{b}}={\left({\mathit{F}}_{\mathrm{b}}^T,{\mathit{M}}_{\mathrm{b}}^T\right)}^T$ between prosthesis and human based on proprioceptive measurements only. The utilized floating base models were first used in space robotics, in order to minimize mutual, disturbing interactions between robot joints and a floating robot base (Dubowsky and Papadopoulos, 1993; Moosavian and Papadopoulos, 2007). In this work, we apply this concept for computing base reaction forces and moments. Figure 10 shows the conceptual approach.

Figure 10.

Schematics for wrench calculation at the prosthesis base, attached to the human body, with F _g being a gravitational force.

${}^{\mathrm{W}}{\mathit{F}}_{\mathrm{b}}$ and ${}^{\phi }{\mathit{M}}_{\mathrm{b}}$ can be derived by considering rows 1 to 6 from (4), (6) and substituting (5) as

$M_{\text{b},\text{b}}(q_{\text{c}})\left(\begin{array}{c}{\ddot{\mathit{r}}}_{\mathrm{b}}\\ {\ddot{\mathit{\phi }}}_{\mathrm{b}}\end{array}\right)+c_{\text{b}}\left(q_{\text{c}},{\stackrel{.}{q}}_{\text{c}}\right)+{\mathit{g}}_{\mathrm{b}}\left({\mathit{q}}_{\mathrm{c}}\right)=\left(\begin{array}{l}{}^{\mathrm{W}}{\mathit{F}}_{\mathrm{b}}\\ {}^{\phi }{\mathit{M}}_{\mathrm{b}}\end{array}\right)+{\mathit{P}}_0{\mathit{J}}^T{\left(\mathrm{C},{\mathit{q}}_{\mathrm{c}}\right)}^{\mathrm{W}}{\mathcal{F}}_{\text{ext}}.$

(32)

The matrix

${\mathit{P}}_0=\left(\begin{array}{ll}{\mathit{I}}_{6\times 6}& {\mathbf{0}}_{6\times m}\end{array}\right)\in {\mathbb{R}}^{6\times 6+m}$

(33)

is sparse with binary entries to adapt the dimension of the Jacobian to only affect the base. For sake of simplicity: in this work, we assume rather slow motion. As time derivatives ${\ddot{\mathit{r}}}_{\mathrm{b}}$ , ${\ddot{\mathit{\phi }}}_{\mathrm{b}}$ , ${\dot{\mathit{r}}}_{\mathrm{b}}$ , ${\dot{\mathit{\phi }}}_{\mathrm{b}}$ are considered to be zero, we obtain a quasi-static model ⁹

$\left(\begin{array}{l}{}^{\mathrm{W}}{\mathit{F}}_{\mathrm{b}}\\ {}^{\phi }{\mathit{M}}_{\mathrm{b}}\end{array}\right)={\mathit{g}}_{\mathrm{b}}\left({\mathit{q}}_{\mathrm{c}}\right)-{\mathit{P}}_0{\mathit{J}}^T{\left(\mathrm{C},{\mathit{q}}_{\mathrm{c}}\right)}^{\mathrm{W}}{\mathcal{F}}_{\text{ext}}$

(34)

after solving for ${}^{\phi }{\mathit{F}}_{\mathrm{b}}$ , ${}^{\phi }{\mathit{M}}_{\mathrm{b}}$ . The contact wrench ${}^{\mathrm{W}}{\mathcal{F}}_{\text{ext}}$ can be estimated (for an arbitrary contact point) via

$\begin{array}{l}{}^{\mathrm{W}}{\widehat{\mathcal{F}}}_{\text{ext}}={\left(\mathit{J}\left(\mathrm{C},{\mathit{q}}_{\mathrm{c}}\right){\mathit{J}}^T\left(\mathrm{C},{\mathit{q}}_{\mathrm{c}}\right)\right)}^{-1}\mathit{J}\left(\mathrm{C},{\mathit{q}}_{\mathrm{c}}\right){\widehat{\mathit{\tau }}}_{\text{ext}}\end{array}$

(35)

$={\mathit{J}}_{\mathrm{C}}^{\#}\left({\mathit{q}}_{\mathrm{c}}\right){\widehat{\mathit{\tau }}}_{\text{ext}},$

(36)

More advanced concepts for force contact point estimations may be used from (Vorndamme and Haddadin, 2021). However, for this work, we rely on this basic version.

Finally, the sought reaction wrench is expressed in the base frame coordinates (in correspondence to a sensor) via

$\left(\begin{array}{l}{}^{\mathrm{b}}{\mathit{F}}_{\mathrm{b}}\\ {}^{\mathrm{b}}{\mathit{M}}_{\mathrm{b}}\end{array}\right)=\left(\begin{array}{l}{}^{\mathrm{b}}{\mathit{A}}_{\mathrm{W}}{}^{\mathrm{W}}{\mathit{F}}_{\mathrm{b}}\\ {}^{\mathrm{b}}{\mathit{A}}_{\mathrm{W}}{\left({\mathit{J}}_{\omega }^T\right)}^{-1\phi }{\mathit{M}}_{\mathrm{b}}\end{array}\right).$

(37)

5.3.1. Practical remarks

Equation (36) allows to calculate a general wrench with respect to any contact point C at (CS)_C, see Figure 7(b). However, the point C has to be chosen carefully in order to allow a general usage of ${}^{\mathrm{W}}{\widehat{\mathcal{F}}}_{\text{ext}}$ . Furthermore, not all components of ${}^{\mathrm{W}}{\widehat{\mathcal{F}}}_{\text{ext}}$ in any configuration of q _c and C may be computable due to singular configurations and numerical reasons. For instance, if the contact point C is chosen to be in the coinciding axes of the wrist (CS)₄, this has the disadvantages (i.e., in $\mathit{q}={\left(\mathrm{0,0,0,0}\right)}^T$ ) that forces along the longitudinal axis of the prosthesis (^W e _x) do not have any effect on the joint torque and result in singular configurations. Furthermore, radial forces parallel to the ground (^W e _y) do not have any affect on the joint torques either, also distorting the results. Consequently, the contact point should be chosen as ${}^4{\mathit{r}}_{\mathrm{C}}={\left(k_c,\mathrm{0,0}\right)}^T$ with k _c ≠ 0. ¹⁰ Furthermore, the following model assumptions are made.

In order to obtain the most stable results for our prosthesis kinematics, we assume the wrench takes the form

${}^{\mathrm{W}}{\widehat{\mathcal{F}}}_{\text{ext}}={\left(\begin{array}{llllll}0& 0& {}^{\mathrm{W}}{\widehat{F}}_{\text{ext},\mathrm{z}}& 0& 0& 0\end{array}\right)}^T$

(38)

which considers the scalar contact force ${}^W{\widehat{F}}_{\text{ext},\mathrm{z}}$ to be parallel to the gravity vector. This is an assumption and other solutions may work. The equations

${\mathit{J}}^{*}\left(\mathrm{C},{\mathit{q}}_{\mathrm{c}}\right)={\mathit{P}}_1\mathit{J}\left(\mathrm{C},{\mathit{q}}_{\mathrm{c}}\right){\mathit{P}}_2^T$

(39)

${\widehat{\mathit{\tau }}}_{\text{ext}}^{*}={\mathit{P}}_2{\widehat{\mathit{\tau }}}_{\text{ext},\mathrm{c}},$

(40)

represent any possible model assumption and are used for selecting sub matrices of J (C, q _c) and rows of ${\widehat{\mathit{\tau }}}_{\text{ext}}$ . P ₁ is a selector matrix for the rows of J (C, q _c), which corresponds to the components of the wrench ${}^{\mathrm{W}}{\widehat{\mathcal{F}}}_{\text{ext}}$ . P ₂ is a selector matrix for the columns of J (C, q _c), determining which sensor information of ${\widehat{\mathit{\tau }}}_{\text{ext},\mathrm{c}}={\left({\mathit{F}}_{\mathrm{b}}^T,{\mathit{M}}_{\mathrm{b}}^T,{\mathit{\tau }}_{\text{ext}}^T\right)}^T$ is used. The selector matrices in our case are chosen to be ${\mathit{P}}_1=\left(\mathrm{0,0,1,0,0,0}\right)$ and ${\mathit{P}}_2=\left({\mathbf{0}}_{m\times 6},{\mathit{I}}_{m\times m}\right)$ . Finally, the estimated wrench follows to be

${\widehat{F}}_{\text{ext},\mathrm{z}}={\mathit{J}}^{*}{\left(\mathrm{C},{\mathit{q}}_{\mathrm{c}}\right)}^{\#}{\widehat{\mathit{\tau }}}_{\text{ext}}^{*}.$

(41)

The aforementioned limitations can be overcome partly by running multiple variants of (39), (40) in parallel. For instance, forces in the direction (^W e _y) might be detected by using only τ₃. Another solution might be an algorithm, which analyses the structure for singularities in realtime for any given configuration of q _c. It then chooses the components of ${\widehat{\mathit{\tau }}}_{\text{ext}}$ and ${}^{\mathrm{W}}{\widehat{\mathcal{F}}}_{\text{ext}}$ (by P ₁ and P ₂) which do not lead to numerical instabilities. Another solution would be the implementation of a suitable force/torque sensor coming with many other difficulties and challenges.

Finally, Figure 11 depicts the entire modeling and control scheme of the ANP.

Figure 11.

Model and control structure of the ANP. More information on the tendon actuation can be bound in Section C in the Appendix. Please also compare to Figure 3 to see the human-inspired mechatronics approach, (Requirements T₃, T₄, T₅, T₆).

6. Experiments

In this section, the validity of the concept is experimentally tested on the ANP prototype. In addition to the experiments, the tests from Table 3 were successfully validated in simulation which may be found in the Appendix Section F. Also, the motion control (i.e., Human-like kinematics) and human-inspired contact response were successfully tested in experiment and may be found in the Appendix Section G. The following experiments focus on body awareness, contact awareness and the grasping validation of the ANP. The utilized model and plant parameters of the ANP may be found in the multimedia material. The experiments were performed by an unimpaired user, the first author of this work, in accordance with the declaration of Helsinki.

6.1. Body awareness

The body awareness is demonstrated by a human user interacting with the ANP, see Figure 12. In this context, also the active backdrivability of the torque controlled actuators is shown. During the experiment, only the floating base gravity compensation ${\mathit{\tau }}_{\mathrm{d}}={\widehat{\mathit{\tau }}}_{\mathrm{g}}\left(\mathit{q},{\mathit{\phi }}_{\mathrm{b}}\right)$ and the underlying joint torque controllers are active. The first row of the figure shows the human tilting the base of the prosthesis, noticeable in the variation of φ _b. Additionally, the prosthesis joints are changed by an interaction wrench ${\widehat{\mathcal{F}}}_{\text{ext}}$ in the gray areas of the figure. Row three shows that $\dot{q}\approx \mathbf{0}$ for the contact-free case, even though q and φ _b are varied and q is not actively controlled—thus metrics (M₁₀) and (M₁₅) are confirmed. Still, q can be changed by an interaction wrench ${\widehat{\mathcal{F}}}_{\text{ext}}$ making the system backdrivable by active control (F₇). The reason for this behavior can be understood by looking at ${\widehat{\mathit{\tau }}}_{\mathrm{g}}\left(\mathit{q},{\mathit{\phi }}_{\mathrm{b}}\right)$ in row four and five. Row six shows the output of the momentum observer ${\widehat{\mathit{\tau }}}_{\text{ext}}$ , which contains the same floating base model. As ${\widehat{\mathit{\tau }}}_{\text{ext}}$ remains essentially constant despite the change in q and φ _b, also showing the goodness of the applied floating base model (M₁₂), (M₁₇). However, an error of about 0.4 Nm can still be observed in ${\widehat{\tau }}_{\text{ext}1}$ which indicates a static model inaccuracy above the admissible error ¹¹ of ${\widehat{e}}_{\tau ,\max }$ , see Table 6. We quantify the goodness of the backdrivability (M₈) and (M₉) by looking at row seven, showing the exchanged energy between human and device. At t = 5 s, the exchanged energy is 0.4 J for deflecting the joints by Δq₁ ≈ 20 deg and Δq₃ ≈ 20 deg.

Figure 12.

Experiment: Body awareness. The prosthesis is tilted by the human while the system is floating base gravity compensated. Consequently, the joints of the prosthesis do not move, unless an intended human contact occurs to move the joints to another position (gray areas), Δ = (113.2, − 38.6, 12.6) deg, {Eval.: F₃, F₆, F₇, F₈, F₉, F₁₁, T₁, T₂, T₃, T₄, T₆, M₉, M₁₀, M₁₂, M₁₃, M₁₅, M₁₆}.

For comparison, an impedance controlled system with a stiffness of 10 Nm/rad would have stored an energy of E_imp = 2 × 0.5 × 10 × (20 × π/180)² = 1.22 J.

The last two rows show the model-based base wrenches, which change in dependency of φ _b, see Figure 12. The black dashed line shows the pure gravity model, while the colored lines are equipped with the estimation from (34), (37) and (41) using the momentum observer ${\widehat{\mathit{\tau }}}_{\text{ext}}$ .

Thus, the system from the figure is also equipped with a drift-free contact awareness, here applied in a contact-free case. As contact and body awareness are almost equal (black and colored lines), apart from the additional noise of the contact awareness, (F₈) and (F₉) can be confirmed by this experiment.

6.2. Contact awareness

The contact awareness of the ANP is used to estimate the contact force ^W F _ext by a human, see Figure 13. Two amplitudes of ${\widehat{\mathit{\tau }}}_{\text{ext}}\approx$ 5 Nm can be observed in row two which are the result of the contact. The observer torque ${\widehat{\mathit{\tau }}}_{\text{ext}}$ is utilized for estimating the external force ^W F _ext in row one by (36). On this basis, the effect on ^b F _b and ^b M _b is calculated by (37), therefore (F₁₀) and (F₁₁) are satisfied.

Figure 13.

Experiment: Contact awareness, {Eval.: F₈, F₁₀, F₁₁, M₁₇, M₁₈}.

Finally, contact awareness is used to monitor the stress in the prosthesis base and to react with a protective control mode, see Figures 14 and 15. In the first experiment (see Figure 14), the prosthesis is programmed to move from q₁ = 0 to q₁ = −π/2. A collision arises from the contact between the prosthesis and the table. The methods for body and contact awareness are used to calculate M _b. M _b increases up to 11 Nm, which may be harmful for a prosthesis user. Figure 28 in the Appendix shows the related joint torques of the ANP which is essentially a collision experiment showing the robustness of the device. The experiment is repeated with activated collision detection and reaction, see Figure 15.

Figure 14.

Experiment: ANP runs into contact without collision detection, {Eval.: F₄, F₁₁, T₅, T₆}.

Figure 15.

Experiment: ANP collides with obstacle with collision detection/reaction, {Eval.: F₉, F₁₀, F₁₁, F₁₂, T₅, T₆, M₁₇, M₁₈}.

After the prosthesis collides with the surface, the base moment ^b M _b exceeds the given threshold r_th = 4 Nm. As a consequence, a control switch from joint impedance control to safe gravity compensation is initiated. The base wrench reduces to ^b M _b ≈ 3 Nm, corresponding to the initial gravity moment only. This experiment validates requirement (F₁₂). For more information, see also the contact awareness simulation in Section F.4 in the Appendix.

6.3. Grasping

In the following two experiments, the overall grasping capabilities of the developed robotic device are validated and we investigate how the novel 4-dof kinematics helps to fulfill complex daily grasping tasks.

The prosthesis is controlled by the user via a graphical user interface (GUI) on a smartphone, see Figure 16, which is further described in Section G.3 in the Appendix. The experimental setup, see Figure 17: Four different objects of daily use are placed on a table at the beginning of the experiment, specifically, a can of tomatoes (250 g), a can of tuna (125 g), a bottle of water (500 g) and a glass (300 g). The aim is to solely use the prosthesis to move all objects from the table top ① to the middle level of the shelf ②, and finally, to the top level of the shelf ③. The three different levels of heights and four different objects where chosen to encourage the operator to use different grasping poses with the prosthesis and thus to exploit the capabilities of the 4-dof kinematics. Thus, in total 24 different orientation changes are required to fulfill all experiments. Another object, a 1 kg bottle, which was grasped by the device, is depicted in Figure 17.

Figure 16.

Unimpaired user controlling the ANP via a GUI.

Figure 17.

Setup of the grasping experiment.

Figure 18 depicts 12 different grasping cases showing the grasping capabilities for different situations. The rows depict the experiments with varying objects (i.e., tomato can (1), tuna can (2), bottle of water (3), and glass (4)). The columns show the different locations (i.e. table top (a), the middle level (b), and the upper level (c) of the shelf). The figure is further analyzed in Section G.4 in the Appendix. Furthermore, the measured data for grasping the can of tuna is depicted and discussed in Section G.5 in the Appendix.

Figure 18.

Grasping validation for differ grasping poses using the Softhand Pro 2 (Della Santina et al., 2018), {Eval.:F₁₃, T₅, M₁₉}.

A final grasping experiment is performed in Figure 19 where the hand of the prosthesis is guided to the object via interaction forces in gravity compensation, instead of using the impedance controller and the smartphone GUI. The related data is depicted in Figure 31 in the Appendix. This experiment shows that the prosthesis can also be controlled by direct kinesthetic teaching. This experiment provides an example of how backdrivability and gravity compensation may be utilized in upper limb prosthetics, apart from achieving enhanced security of a low impedance prosthesis and simulating human limb behavior. Further information may be found in Section G.6 in the Appendix. Consequently, the grasping is evaluated in this final experiment (M₁₉).

Figure 19.

Experiment: guiding the prosthesis via interaction forces to grasp an object, {Eval.: F₆, F₁₃, T₂, T₃, M₁₉}.

7. Discussion

In this section, the prosthesis capability, the design paradigm and the mechatronic device are discussed.

7.1. Prosthesis capability

In this work, novel human-inspired capabilities for upper-limb prostheses such as body awareness, contact awareness and human-like contact response were proposed, see Figure 1. It is hard to straight forward compare these capabilities with traditional prosthesis control methods such as velocity-based sEMG control (Lenzi et al., 2016), or human machine interfaces for prostheses (Resnik et al., 2014). Rather, our envisioned capabilities are better characterized as a novel system behavior or joint-torque-level autonomy, aiming to recreate key functionalities of the human neuromuscular system, such that the prosthesis can mechanically behave similar to a natural human arm if controlled appropriately. With this, the prosthesis shall autonomously react to and compensate for environmental influences, such as orientation changes or interaction forces, in particular without requiring additional user commands. Therefore, our approach is rather designed to coexist with or even complement widespread prosthesis control methods such as velocity-based sEMG control interfaces.

Our first prototype served the introduction of fundamental soft and tactile robotics modeling and control concepts to prosthetics. Following our novel design paradigm, body awareness, contact awareness and human-like contact response were realized by the simplest possible implementation. The contact awareness can, for example, handle both contact and non-contact situations well. However, due to the underlying observer system, the contact awareness produces more noise than the body awareness. Another trade off in the current design is that all prostheses joints require additional torque sensing. This significantly impacts the mechanical design approach, requiring space and weight, which otherwise could be used to optimize the prosthesis towards higher joint torques, speed or battery capacity. Still, traditional design approaches are known to not be able to render the human-like capability we target for. Also, the modeling and control approaches require more computation power in comparison to standard low-complexity control methods. However, given the current computational power, this is not a real limitation anymore, even for state-of-the-art mobile processors.

Clearly, there is still quite a gap between the current capability of the ANP and the antetype human body. Numerous improvements will follow in future works: (i) the extension of body and contact awareness and human-like contact response to the prosthesis hand, (ii) the ability to recognize multiple force directions and contact forces via proprioceptive measurements (Vorndamme and Haddadin, 2021), (iii) adding tactile skins (Mittendorfer and Cheng, 2011) integrated within the concept of body and contact awareness, (iv) develop a tactile feedback system for the prosthesis user using proprioceptive sensor information, (v) sophisticated human-inspired control laws for context-informed impedance adaptation and learning, (vi) the integration of aforementioned concepts into state-of-the-art prosthesis control methods such as velocity-based sEMG control, (vii) the upgrade of body and contact awareness on acceleration and velocity level.

7.2. User benefits

The benefits of the new prosthesis paradigm for future users are expected to be:

1. Increased mechanical safety: (i) impacts can be absorbed by the compliantly controlled elastic joints, and (ii) the risk for accidents or falls can be reduced as the device adapts to the environment due to its compliance. For example, a standard setting of the prosthesis could be a low joint stiffness of K_imp,i = {0… 5 Nm/rad}. Only, if the joints are actively controlled (e.g., ${\dot{\mathit{q}}}_{\mathrm{d}}\ne 0$ ), joint stiffness is increased to, for example, K_imp,i = 50 Nm/rad.

2. The human-like 4-dof kinematics allows to render human dexterity. Consequently, complex grasping tasks in restricted environments can be performed. The kinematics can be fully exploited once powerful HMIs are integrated.

3. Human-like motion and dynamics, based on joint impedance control, mimics the natural human limb in terms of inertia, stiffness, damping and contact response behavior. It systemically renders mechanical properties of human muscles and tendons closer than rigidly controlled prostheses, allowing more natural and fluid movements and control. Furthermore, less vibrations occur in comparison to position controlled devices (Ott, 2008; Vorndamme et al., 2016).

4. Low-level semi-autonomy: the prosthesis mimics natural capability of the human body such as gravity compensation or body and contact awareness. Consequently, the prosthesis is expected to behave closer to a human limb and therefore may better meet human expectations.

5. Proprioception: the prosthesis realizes a proprioceptive contact force measurement system using the joint torque sensors and model-based observers. These can be used as sensor for a haptic user feedback system in the future.

6. Guide and control via contact forces: the kinesthetic guidance via direct interaction forces allows to effortlessly adjust large orientation changes of the device. This feature may fulfill its full potential when joint positions/full configurations can be seamlessly stored and interpolated, or certain joints can be software locked. With this, even the functionality of some passive wrists (Cappello et al., 2023; Montagnani et al., 2013, 2017) can be reproduced by active control.

7. Socket protection: the contact awareness monitors and protects the residual limb attached to the prosthesis socket by suitable reflex control switching to gravity compensation or more sophisticated reflexes.

8. More functionality through high-level controllers: humans use stiffness and damping control for more efficient interaction with the environment and grasping. Modern human-inspired learning and adaptation algorithms have the potential to improve grasping and interaction performance.

In the following, the design approach and it implications is further discussed.

7.3. Design paradigm

The proposed sim2real-guided and human-inspired design paradigm provides two novel perspectives for prosthesis design. (i) The human-inspired design approach extends the degrees of freedom to realize more human-like behaviors in prosthetics. In this work, fundamental feedback and feed-forward algorithms have been developed. Future work may cover closer biological correspondences of human motor control, for example, by modeling the systemic functionality of human neural circuitry (Kuehn and Haddadin, 2017). (ii) The sim2real-guided design paradigm translates robotics modeling, design and control approaches to prostheses and builds on best practices in model-based design. The next section deals with the discussion on the prosthesis mechatronics.

7.4. Prosthesis mechatronics

In this work a mechatronic solution with close to human-like size and weight is introduced. As of today, many components necessary for such a device are not available on the market and advanced custom solutions had to be designed, showing the necessary design thinking in the mechatronics design.

Furthermore, state-of-the-art mobile processors are suitable for computing all necessary models, observers and controllers. At the moment, the prosthesis provides sufficient torque for lifting payloads up to 1 kg. A key objective for future versions should be to achieve human-level strength and speed. Currently, not all desired human motions can be executed in terms of wrist speed (see Table 8) and maximum torque. For the wrist design, a solution was found which allows joint-torque sensing combined with several degrees of freedom at close to human-level compactness. This is possible because the tendon stroke can be wrapped around the spool and the torque sensor is virtualized with the tendon force measurements. Yet, a set of stronger tendons (Toedtheide et al., 2021) seems necessary due to some failures at full loads after several operating hours. For performing systematic user testing, the current Technology Readiness Level (TRL) has to be systematically increased on the electronics level.

Overall, the presented first generation ANP has also principal limitations. For example, batteries and internal computation unit for the joint-level and coordination controllers are not integrated yet. Joint 1 and 2 and the electrical circuit boards are to be further minimized for providing additional design space. More specifically, let us assume a battery with a volume of 140 cm³ located in the prosthesis (e.g., a diameter of 63 mm and a height of 45 mm) and a second one of the same volume located at the prosthesis socket. This configuration could provide 3.3 h of pure load, 6.7 h of idle operation and 5.5 h considering a 20/80 profile of load/idle. The weight of the batteries would be 0.5 kg. ¹² Alternatively, the battery could be fixed to the user belt. After some electronics iterations and standard mechanics robustification measures, the device is sufficiently mature for prosthesis applications outside the lab. Note that the fundamental technology is related to those established in state-of-the-art soft and tactile robotics (Haddadin et al., 2022) for which industry-standard and medical device regulation robustness was achieved on product level. The KUKA LBR Med, for example, shows that medical certifications and standards can be met with an advanced mechatronics concept (KUKA AG, 2023). Future hardware development will focus on further weight, size, robustness and joint torque optimization, clean cable guidance, housing, human machine interfacing and advanced task-level controllers.

7.5. Experiments

To date, the ANP has been tested under laboratory conditions in a mechatronic test-bench and by a single human expert user, the first author of this work, validating the specified capability such as body awareness, contact awareness, human-like kinematics, human-like contact response, and grasping. The focus of the experiments is on basic technological capability demonstration and validation for the ANP, and in general, exemplary for a device developed by the sim2real-guided design approach. In this context, the sole purpose of the expert user is to serve as a floating base and to produce typical human-induced control data for evaluating the device performance and demonstrating that execution of complex (grasping) tasks is possible. This includes typical trunk and upper limb motions as well as control inputs, which cannot be obtained on a pure technical test-bench. These tests are essential as complex physical reactions of close-to-real use cases can be investigated. In essence, the experimental results validate the proposed paradigm shift and show its readiness to be fused with state-of-the-art human-machine interfaces. Due to the given scope of research and the methods of evaluation (i.e., via the metrics M in Table 3), any potential bias on the part of the human expert user is expected to have a negligible effect on the scientific results. A bias generally occurs for studies on human behavior, which is not the case for this work. The floating base and tactile capabilities of the prosthesis, shown in Figures 12 and 19, are in fact mechatronic compensations which do not provide a favorable mode of operation or interaction, which a biased user could take advantage of. For Figure 18, the grasping experiment, device was capable of performing versatile grasping tasks according to M₁₉ in Table 3. This metrics is true regardless of the pilot and its potential bias. Consequently, a user bias is expected to be insignificant for this research. In future work, the next system generation with higher TRL will be validated with impaired users.

Note that due to the focus in current experiments on principal capability validation and verification, no statement about user acceptance, ergonomics or true quality of human-device interaction of amputees is yet possible. As a single user experiment suffices these goals, statistically relevant data was not necessary to be generated and is thus also not meaningful at the current stage.

For testing and validating the new paradigm with amputees, several steps are still necessary: (i) The robustness, safety, and overall TRL of the next generation device need to be increased by the measures described in the mechatronics design paragraph. (ii) The socket of the impaired pilot, a battery and a computation unit need to be integrated into the prosthesis. (iii) An sEMG-based control-interface needs to be integrated. (iv) The proposed key capabilities from Figure 1 need to be integrated into an HMI concept. Currently, for example, the joint stiffness adaptation is adjusted via a smartphone GUI. This needs to be connected to user inputs as described in the user benefits paragraph. Furthermore, other capabilities such as storing key points and interpolating motions, learning of entire manipulation skills, and the active control of passive wrist behaviors (Cappello et al., 2023) can further increase the versatility and benefits for prosthesis users. (v) Suitable experiments and metrics need to be defined to fully evaluate the proposed capabilities from Figure 1 (vi) Also, for performing human trials with impaired users and subsequent clinical evaluation, the next generation device needs to go through full ethics approval with underlying risk analysis and mitigation, safety-rated development, and development according to certified quality processes.

Once aforementioned steps are fulfilled, the next generation device will be tested in statistically relevant user trials with impaired users.

7.6. Most notable results

Table 7 summarizes the main experiments and relates them to the initial feature requirements from Figure 1.

Table 7.

Summary of Results.

Feature	Expected behavior	Figure
Body awareness	Computation of all gravitational joint torques and base wrenches for non-contact situations; hold position using the floating base gravity compensation	12
Contact awareness	Computation of all gravitational joint torques and base wrenches for contact and non-contact situations	13
	Estimate the effect of a contact force at the residual limb; minimize the wrench at the residual limb by switching to gravity compensation in case of a collision if the observed wrench is exceeded	15
Human-like contact response	Compliant mechanical behavior which can be applied online in software	25
	Active backdrivability of joints (i.e., zero stiffness and damping) enabling kinesthetic guidance via interaction forces	12
Human-like kinematics	Grasp objects at various locations by suitably exploiting the 4-dof prosthesis kinematics	18
Body awareness & Human-like contact response	Use body awareness and human-like contact response with zero stiffness to guide the prosthesis via interaction forces to the desired grasping pose	19

8. Conclusion

In this paper, we introduced a novel prosthesis class aiming to render key capabilities of the human neuromuscular system and hereby contributed via (i) novel prosthesis capabilities, (ii) a new robotic prosthesis device, and (iii) a novel prosthesis design paradigm. We introduced the Artificial Neuromuscular Prosthesis ANP, which imitates behavioral aspects of the human body in terms of its (i) body awareness, (ii) contact awareness, (iii) human-like contact response, and (iv) human-like kinematics as defined in the framework of this paper. Consequently, the ANP behaves mechanically similar to a human arm—in the sense of our target anthropomorphic design specifications. To achieve this goal, we introduced a novel development paradigm for upper-limb prostheses, which is characterized by (i) a biologically inspired mechatronic solution, and (ii) the subsequent rigorous specification, mathematical modeling, control, simulation, design and evaluation of the prosthetic device. We presented the novel custom-developed ANP mechatronics, which consists of a four degrees-of-freedom elbow-to-wrist kinematics, joint-torque controlled actuators, a tendon-driven wrist, an IMU for spatial orientation sensing—at size and weight of 1.7 kg without hand and battery. We introduced soft and tactile prosthesis control methods for the ANP, consisting of a floating base gravity compensation, joint-level impedance control with damping design, and momentum observation for contact wrench monitoring - all of them being new to the prosthesis world - to render the desired functionalities of the human archetype.

With this work, we demonstrated that the latest concepts from robot mechatronics known from soft and tactile and humanoid robotics can indeed be translated to upper-limb prostheses and promising solutions in terms of weight, size and overall capability are found. In this regard, future steps for battery integration were discussed. We also detailed how such a complex and advanced design can be realized in just a single prototype generation with our simulation-guided design approach. Overall, a significant technological leap towards more human-like robotic prostheses could be achieved.

In dynamic simulations and real-world experiments, we could validate all desired prosthesis functionalities such as (i) body awareness, (ii) contact awareness, (iii) human-like contact response, (iv) human-like kinematics and (v) grasping as mathematically defined in this work. The simulations turned out to be highly eligible for the model-based design and in very good agreement with the experimental system behavior.

While the system mechatronics and its controllers were rigorously validated, future work focuses on developing the next generation to improve the TRL, robustness, weight, size, torque density, speed, clean cable guidance, housing, human machine interfacing and more advanced task-level HMIs and controllers.

With this, user-centered and even clinical studies will be performed, aiming to cover a statistically relevant number of participants. These studies should focus on (i) user acceptance, (ii) device ergonomics and (iii) the human-device-interaction using state-of-the-art prosthesis human machine interfaces.

Supplemental Material

Supplemental Material

Supplemental Material

Supplemental Material