Sage Journals: Discover world-class research

Abstract

Manipulators based on soft robotic technologies exhibit compliance and dexterity which ensures safe human–robot interaction. This article is a novel attempt at exploiting these desirable properties to develop a manipulator for an assistive application, in particular, a shower arm to assist the elderly in the bathing task. The overall vision for the soft manipulator is to concatenate three modules in a serial manner such that (i) the proximal segment is made up of cable-based actuation to compensate for gravitational effects and (ii) the central and distal segments are made up of hybrid actuation to autonomously reach delicate body parts to perform the main tasks related to bathing. The role of the latter modules is crucial to the application of the system in the bathing task; however, it is a nontrivial challenge to develop a robust and controllable hybrid actuated system with advanced manipulation capabilities and hence, the focus of this article. We first introduce our design and experimentally characterize its functionalities, which include elongation, shortening, omnidirectional bending. Next, we propose a control concept capable of solving the inverse kinetics problem using multiagent reinforcement learning to exploit these functionalities despite high dimensionality and redundancy. We demonstrate the effectiveness of the design and control of this module by demonstrating an open-loop task space control where it successfully moves through an asymmetric 3-D trajectory sampled at 12 points with an average reaching accuracy of 0.79 cm ± 0.18 cm. Our quantitative experimental results present a promising step toward the development of the soft manipulator eventually contributing to the advancement of soft robotics.

Keywords

Assistive robotics soft robotics machine learning robot control

Introduction

Activities of daily living (ADLs) refer to the daily tasks people should be able to independently accomplish¹ and are grouped into two general categories: (i) basic ADLs (BADLs) and (ii) instrumental ADLs. However, achieving these is a significant challenge for the elderly generation with chronic ailments and loss of abilities.^2,3 Initially, children took care of their elders; however, nowadays the majority of the population is dependent upon professional healthcare services. This has brought about an interest to the development of advanced assistive technical devices (also referred to as assistive robotics) in order to promote independent living and consequently provide the possibility to the elderly generation to improve their quality of life.^2,4,5 This work focuses on the development of assistive devices for BADLs, in particular, the task of bathing which is considered challenging because the entire body is exposed in this scenario. Literature reveals, that this area has been scarcely investigated, with only two commercially available technical solutions.

Oasis Seated Shower system:⁶ provides an independent, safe, and hygienic shower experience. The system is equipped with a push-button soap application and rinse system. On the downside, it completely lacks active interaction between the user and the tool which also includes lack of functionalities such as rubbing/wiping the senior. Furthermore, it is bulky and expensive.

Seat Lift Device:⁷ is much simpler, partially automated, and helps positioning the user in the bathing task. However, it does not provide any assistance in showering task.

There still exists a multitude of unaddressed issues in this domain where the system should: (i) be client-oriented i.e. translate customer requests into technical requirements; (ii) be safe, reliable, and adaptable for human-robot interaction;^8,9 (iii) be highly dexterous with a large reachable workspace; and finally (iv) autonomously reach difficult and sensitive regions of the body (refer to Figure 1).

Figure 1.

General overview of the bathing task. The soft manipulator approaches the back region of the user, one of the most critical and difficult regions to reach. The lower limbs have also been identified as difficult task. An additional soft manipulator could be mounted on the wall for the lower limbs.

Soft robotic manipulators

In order to address the above limitations and increase the user autonomy at home, a suitable approach is to exploit principles and technologies from the rapidly growing research field of Soft Robotics¹⁰ for developing a manipulator as a shower arm in the bathing task. Soft robotics refers to a new generation of flexible robots that take inspiration from muscular hydrostats such as octopus tentacles, elephant trunks, etc., in the use of soft materials to achieve advanced manipulation capabilities while being safe to interact with.^11
–
13 These natural manipulators have inspired the development of novel soft robotic manipulators for obtaining a desired gamut of motion. These manipulators comprise of actuators elements that can be arranged into the single body or repeated into single identical modules. The advantages of the modular approach to soft robots, introduced by Onal and Rus¹⁴ for the first time, demonstrates how actuation units can be composed in various, possibly redundant arrangements to achieve exceedingly complex motions. Taking advantage of this achievement, a mechanical modular manipulator can be designed by concatenating modular blocks, which, in turn, are an assembly of soft actuators that can be intrinsic (actuators are located within the body thus forming part of the animated mechanism), extrinsic (the actuator is remote and motion is transferred into the mechanism via a mechanical linkage), or hybrid (combination of intrinsic and extrinsic actuation).¹⁵

We have considered that the technical requirements for the design of a soft manipulator are defined by the workspace that it should cover, which includes a volumetric space around and close to the users’ body as shown in Figure 1. This will be translated into a mechanical modular manipulator by concatenating three modular blocks such that (i) the proximal module is a structural module based on radially arranged cables that are used to compensate for gravitational effects and (ii) the centre and the distal modules on hybrid actuation are identical functional modules. The success of the manipulator is highly dependent upon the functional capabilities and working of the hybrid actuated module which is a nontrivial challenge. Consequently, the focus of this work lies in the development of a reliable and controllable distal module.

Challenges in design

The challenge of developing manipulators that rely on the combination of fluidic actuators and cables i.e. hybrid actuation has been largely faced in the surgical robotics field opening the door to other applications.¹⁶ Good examples of soft manipulators based on tendons pulled in relation to pressurized chambers are: the KSI tentacle manipulator powered by a hybrid system of pneumatic bellows and electric motors,¹⁷ the continuum robot proposed by Pritts¹⁹ uses 6-8 contracting/extending McKibben actuators for providing two-axis bending, the Air-Octor manipulator that uses three cables and a single chamber,²⁰ and the OCTARM continuum robot²¹ that is able to perform adaptive manipulation in challenging environments by using air muscle actuators. These examples have been integrated into a more comprehensive literature analysis, as reported in Table 1, that provides a chronological order of some examples for bioinspired manipulators that cover all three categories focusing on the actuation strategy that enables specific functionalities (i.e. elongation/contraction, bending and variable stiffness). This table also indicates the major achievements in Soft Robotics concerning the design of manipulators based on variable stiffness mechanisms. Moreover the Application field of Table 1 highlights how none of these is intended for assistive robotics. It is clear how our proposal of using Soft Robotics methodologies and mechanisms for building the first assistive soft robot is challenging and completely new. In order to develop a manipulator as a shower arm in the bathing task, we investigate an actuation strategy that enables all the mentioned specific functionalities (i.e. elongation/contraction, bending and variable stiffness).

Table 1.

A summary of the design and control of some examples of soft robotic manipulators.

Manipulator	Publication Timeline	Application	Design	Actuation		Functionalities	Control
Manipulator	Publication Timeline	Application	Design	Type	Actuators	Functionalities	Control
KSI tentacle¹⁷	1995	Endoscopy Agriculture Nuclear	Modular	Hybrid	Bellow-shaped pneumatic bladder Radially arranged cables	Variable stiffness Shortening Elongation Omnidirectional bending	Model based
Active Hose¹⁸	2001	Search and Rescue	Modular	Intrinsic	Wounded tube actuator	Omnidirectional bending	—
Artificial muscle continuum robot¹⁹	2004	NSS*	Modular	Intrinsic	(Extensor + contracting) pneumatic Chambers	Omnidirectional bending	—
Air-Octor²⁰	2005	Search and rescue	Modular	Hybrid	Single pneumatic chamber Radially arranged cables	Variable stiffness Shortening Elongation Omnidirectional bending	—
Oct Arm²¹	2005–2008	Industrial Space Military	Modular	Intrinsic	Radially arranged (Extensor + Contracting) pneumatics chambers	Shortening Elongation Omnidirectional bending	Model based²² Hybrid²³
Octopus²⁴	2010–2016	Marine	Single body	Extrinsic	Cable	Elongation Bending	Model based Learning based
BHA²⁵	2011–2016	Industrial	Modular	Intrinsic	Radially arranged bellow-shaped pneumatics chambers	Elongation Omnidirectional bending	Learning based²⁶
Highly articulate manipulator enabled by jamming media²⁷	2012	NSS*	Modular	Hybrid	Granular jamming Radially arranged cables Springs	Variable stiffness Shortening Omnidirectional bending	—
Shrinkable, stiffness-controllable soft manipulator²⁸	2014	NSS*	Modular	Hybrid	Latex chamber Radially arranged cables Braided fabric	Variable stiffness Shortening Elongation Omnidirectional bending	—
STIIFF–FLOP^29,30	2014–2016	Surgery	Modular	Intrinsic	Radially arranged pneumatics chambers Granular jamming	Variable stiffness Elongation Omnidirectional bending	—
STIIFF–FLOP^29,30	2014–2016	Surgery	Modular	Hybrid	Radially arranged pneumatics Radially arranged cables	Variable stiffness Shortening Elongation Omnidirectional bending	—
Multi-segment soft fluidic actuator³¹	2015	NSS*	Modular	Intrinsic	Anisotropic pneumatic chambers	Omnidirectional bending	Model-based

*NSS: not specifically stated.

**Simulation.

Challenges in control

Soft robots often lack an analytical model due to their nonlinear material properties and high-dimensional articulated structure. This makes predictable and controlled movements with positional accuracy a significantly challenging task. Consequently, one of the open research questions is to achieve task space control that allows these robotic systems to accurately follow a trajectory, that is, point-to-point motion control. This is essential for a bathing task where close contact is required between the manipulator and the human to perform scrubbing and cleaning. The objective of point-to-point motion control is to find the correct combination of input actuation at each discrete sample, that is, inverse kinematics/kinetics. Immega and Antonelli¹⁷ proposed a cascaded spline arc approach that controls the cables of the KSI tentacle that roughly positions it in 3-D space, which is further improved through a closed-loop vision feedback. Giorelli et al.²⁴ used a Jacobian based approach to reach an average tip accuracy of 6% of the total manipulator length. Marchese et al.³¹ applied a closed-loop controller on a 3D soft-arm to position the end effector to reach a ball with a diameter of 0.04m. These traditional model-based methods are limited in accuracy and computational costs by modelling assumptions that need to be improved for practical application of soft manipulators. Learning based control^23,24,26 is a promising approach to automate low-level sensorimotor skills. The underlying principle is to allow the manipulator to autonomously explore its environment and correlate sensory and motor spaces. This work investigates the application of these methods for the previously unaddressed challenge of the control of hybrid actuated systems.

The paper is organized as follows: Section ‘Design’ presents the design, fabrication, and characterization of the single distal module. Section ‘Control Framework’ presents the inverse kinematics control approach that is incorporated as a low-level module in the hierarchical controller presented in Section ‘Hierarchical Controller’ Experimental methodology and results in point-to-point motion control is reported in Section ‘Experiments’ followed by a comprehensive discussion of the main outcomes of the proposed work is stated in the last section of the paper.

Design

The overall design of the distal segment relies on the total length which should be able to reach the target workspace which includes back and/or lower limbs (refer to Figure 1). On the one hand, a McKibben-based actuation maximizes the reachable length due to the elongation performances of the fluidic actuators, thus, reducing the number of modules required to achieve the desired length. However, it is unstable due to gravitational effects. On the other hand, the use of cables enables multiple functionalities of shortening, omnidirectional bending, and compensation for gravity effects. However, it requires a higher number of modules for covering the same length. We propose to embed both technologies within the distal module in order to complement the limitations of each other while increasing the overall functionality of the complete module. Thus, the distal module is hybrid actuated. The innovative element in its mechanical design is represented by the flexible fluidic actuators that are McKibben based but with bellow-shaped surface, discussed subsequently.

Design and manufacturing of the distal module

Three McKibben actuators and three cables are alternately displaced at an angle of 60° along a circle with a radius of 3 cm. A central hollow chamber of 1.4 cm diameter has been inserted to facilitate the flow of water and soap. The total length of the module is 20.5 cm, while the total weight is 180 g (refer to Figure 2). The cables and chambers are decoupled, meaning that each one has a dedicated activation line for tension and pressure regulation, respectively.

Figure 2.

Flowchart showing the fabrication process for the single McKibben actuator.

Design of the McKibben actuator

The manufacturing process for the McKibben actuator^32–34 comprises of the following steps: (i) forming the external chamber: A braided sheath (Pro Power PETBK3B10, Farnell Components) with a length of 60 cm and internal nominal diameter of 0.3 cm is expanded and inserted on a metallic cylinder with a diameter of 0.8 cm. A mechanical deformation is applied in the inferior/superior direction to introduce a fiber deformation until a cylindrical bellow structure is created. The final shape is “memorized” by applying a cyclic uniform heat along the structure after which the actuator has an outer diameter of 1 cm and a total length of 19.5 cm (refer to Figure 2). A recommended temperature is around 350°C for 3 min. (ii) Combining the internal and external chambers: A balloon (latex) is inserted into the braided sheath. It is delimited at both ends by endcaps which provide an air-source from one end and an anchoring point to the lateral surface from the other. The bellow-shaped external chamber facilitates a radial containment effect of the internal elastic chamber, thus, improving the elongation performances of the combined structure. (iii) Structural support: The entire module is completed by adding a custom-made helicoidally shaped structure along its length (Figure 3). It houses the three cables, the three chambers, and the internal channel, thus, providing a constraint for the structural elements and avoiding lateral expansions of the actuators. The last part of the module consists of the lateral interfaces (end plates) for confining the module.

Figure 3.

CAD design of the single module; on the top right a section view of the module with the three McKibben-based actuators (identified by A1–A2–A3), the three cables (identified by B1–B2–B3), and the internal channel for the provision of water.

Kinetic characterization of the distal module

The kinetic characterization refers to the evaluation of the complete behavior of the system that includes contraction/elongation, bending/stiffness, and the complete reachable workspace, as discussed in the subsequent subsections.

Experimental setup

The experimental set-up for the characterization of the distal module consists of: (1) a rigid rectangular frame made of 0.8 cm acrylic sheet that orients the module vertically downward; it is custom designed to encompass the actuation systems; (2) Pneumatic Set-up: three proportional pressure-controlled electronic valves (K8P Series by EVP Systems, Italy, Input: 0 - 10 V, Output: 0 -3 bar), one filter (EVP Systems, Italy: MC-104FB0); one manometer (EVP Systems, Italy: M043-p12 0-12bar); one stand-alone air compressor; (3) Cable Set-up: three Hs-645 Hitech Servomotors; (4) A six-DOF electromagnetic sensing probe (Aurora® Tracking by Northern Digital Inc., Canada) has been fixed at the tip of the module base which is capable of sub-millimetre position tracking. This is a visualization tool that provides a log of tracking movements at the end of an experiment (Figure 4).

Figure 4.

Experimental setup for the single module characterization.

Elongation/contraction

The linear motion of the module was captured by moving it from a completely contracted state to a completely extended state. The contracting state corresponds to the cables pulled by the servomotors without the chambers being inflated. All the servomotors are then simultaneously rotated through 180° in steps of 20°. The position obtained after the completion of this rotation corresponds to the neutral point of the manipulator. Next, the pneumatic chambers are inflated from 0.5 to 1 bar. As explained in,³⁵ we start from 0.5 bar as it is the inferior pressure limit of a single actuator i.e., below this limit the actuator does not detect any global elongation movement. Figure 5 illustrates the displacement of the module along the z-axis from the contracted state to the extended state which is found to be 5 cm. With respect to the neutral position, this implies that there is a contraction of 3 cm and elongation of 2 cm.

Figure 5.

Elongation–contraction module characterization. The blue marker represents the starting configuration without active elements.

Bending and stiffening

In order to perform the bending tests, we evaluated two different activation patterns: (i) bending due to a single cable and single chamber: a single randomly chosen cable of the system is contracted from an untensioned state into a tensioned state by rotating the servomotor from the neutral position through 180° in incremental steps of 20°. When the maximum tensioned state is achieved, the diametrically opposite pneumatic chamber is inflated by applying a pressure from 0.5 to 1 bar, with an incremental step of 0.1 bar. Referring to Figure 3, this activation pattern is achieved when A1 collaborates with B3, A2 with B1, or A3 with B2. Figure 6 illustrates the resulting behavior from this activation pattern. The global bending of the system contributed by the cable undergoes a displacement of 60 mm along the z-axis (viewed as the xz-view and yz-view in Figure 6). A stiffness variation can be observed when pneumatic actuation begins which is due to an antagonistic effect. (ii) Bending due to two cables and two chambers: two adjacent cables of the system are contracted from an untensioned state into a tensioned state by rotating the corresponding servomotors simultaneously through 180° in incremental steps of 20°. Referring to Figure 3, this activation pattern can be achieved when first B1–B2 or B2–B3 or B3–B1 are activated. When the maximum tensioned state is achieved, the diametrically opposite pneumatic chambers corresponding to each servomotor, that is, A2–A3, A3–A1, or A1–A2, respectively, are inflated similar to the procedure explained previously. Figure 7 depicts resulting behavior from this activation pattern. The global bending of the system contributed by the cables undergoes a displacement of 5 cm along the z-axis. The reduced value in comparison to the first activation pattern is due to the double-cable activation that pulls the cables with an increased force. It can also be concluded from these figures that the chambers contribute to adapt the tip orientation of the module. The hybrid actuation is promising because of the stiffness variation for applying restrained forces in the interaction with the user.

Figure 6.

Activation pattern with a single cable (B3 in Figure 3) up to the maximum position and activation of the opposite chamber (A1 in Figure 3). (a) Global bending movement of the element; (b) xz view; (c) yz view. Arrows with different colors show the orientation of the module: x, y, z directions in green, blue, and red, respectively. Arrows with different colors show the orientation of the module: x, y, z directions in green, blue, and red, respectively.

Figure 7.

Activation pattern with two cables (B1–B2 in Figure 3) up to the maximum position and activation of the two opposite chambers (A2–A3 in Figure 3). (a) Global bending movement of the element. (b) xz view. (c) yz view. Arrows with different colors show the orientation of the module: x, y, z directions in green, blue, and red, respectively. Arrows with different colors show the orientation of the module: x, y, z directions in green, blue, and red, respectively.

Reachable workspace

The workspace measurement (see Figure 8) aims to measure all the reachable positions of the module tip. A total of 8000 points were recorded which resulted from a permutation of all six inputs applied using the following values: (i) servomotors: 0°, 45°, 90°, 135°, 180°; (ii) pneumatic pressure: 0 bar, 0.5 bar, 0.8 bar, and 1 bar.

Figure 8.

(a) Workspace evaluation for the single module. The blue point represents the initial position of the module in an unactivated state. (b) Workspace view from the X–Y plane. (c) Workspace view from the X–Z plane. (d) Workspace view from the Y–Z plane.

Figure 8(a) depicts the isometric view of the resulting point cloud, highlighting the workspace in the range of 21.5 cm × 17.2 cm × 10 cm in the x–y–z axes, respectively. The blue marker corresponds to the starting configuration of the end effector. The overall shape of the workspace can be categorized as a volumetric convex with the lower limits defined by the maximum bending capabilities (approximately 90°) of the module, as expected. The asymmetry of the shape can be credited to the manufacturing procedure i.e. the irregularities that are introduced during the assembly phase which are not easily identifiable when the module is at rest. For a detailed analysis, the workspace is also presented from a 2D viewpoint from all three planes. Figure 8b shows the top view of the workspace from the X-Y plane. The shape is circular, as theoretically expected. Figure 8c indicates an asymmetric response of the system from the X-Z planar viewpoint, whereas a symmetric response of the system from the Y-Z planar viewpoint is depicted in Figure 8d. Data points are more pronounced toward the left portion in the former graph as a result of a tightened cable attachment to the respective servo arm.

Control framework

Learning based controllers^36,37 offer a promising solution to address the control challenge. In particular, we take inspiration from a previous work³⁸ where each actuator is considered an autonomous agent that resides within and shares an environment forming a distributed multi-agent system (MAS).³⁹ The underlying objective is to enable the agents to coordinate their actions to learn a joint optimum behaviour. Reinforcement Learning (RL)⁴⁰ is a very attractive online optimization technique in milieu of automating a solution for MAS as it assists in (i) model-free learning and (ii) online learning. The former is beneficial to learn optimal behaviour in unpredictable environments where the dynamics are not known a-priori, whereas, the latter is particularly useful to learn behaviour without specifying the underlying actuation technology of the system. This, however, requires that the agents learn in an independent manner⁴¹ similar to the Iterated Prisoner’s Dilemma.⁴²

Reinforcement learning preliminaries

RL is a sequential decision-making tool where an agent iteratively interacts with an environment modeled as a Markov decision process (MDP). An MDP is a 4-tuple <S, A, T, R>, where S is the state space, A is the action space, T is the environment dynamics given by the conditional probability P_r {s′ | s, a} that taking action a in state s will lead to new state s′, and R is the reward received when an action a in state s leads to a new state s′. An agent acts on the environment by selecting actions according to a policy π, that is, a mapping from states to actions. Starting from an initial state, the agent iteratively interacts with the MDP according to the policy to generate a trajectory which refers to a sequence of state-action-reward tuples $< s_{0} \to a_{0} \to r_{1} \to s_{1} \to a_{1} \to r_{2} \to s_{2} \to a_{2}, \dots >$ . The return over an infinite horizon trajectory refers to the cumulative discounted reward as $R_{t} = \sum_{k = 0}^{\infty} γ^{t} r_{k + t + 1}$ , where $γ \in [0, 1)$ , and r is the reward received at each time-step. This can also be extended to a finite horizon setting for episodic tasks where γ = 1. In the context of control, an action-value function (the utility is described using the action-value function as opposed to the more commonly used value-function $V^{π} (s) = E [R_{t} | s_{t} = s]$ as a matter of convenience to the underlying work) Q is widely employed to evaluate the utility of executing an action a in a state s and following a given policy π thereafter, given as⁴³

Q (s, a) = E {\sum_{k = 0}^{τ} γ^{t} r_{k + t + 1} | s_{t} = s, a_{t} = a, π}

where τ is the terminating condition, s is the state of the system at time t, and a is the action selected in state s at time t in given policy π. This work considers trajectories that finish within a finite horizon; hence, τ represents when either the goal s_g is reached or a maximum number of action trials are attempted in an episode. In optimal control problems, the goal is to find an optimal control policy π* that maximizes the expected cumulative discounted return regardless of the initial state. The focus of this work is to derive π* via policy iteration (PI),⁴⁴ one of the two main classes of dynamic programing to solve MDPs.

Formally, PI is a subroutine that runs trajectories iteratively for a given duration, wherein it alternates the processes of policy evaluation and policy improvement as follows: Starting from any initial position at time-step t, an action in the current state is chosen according to a policy π _t and is evaluated by Q_t . In the next time-step t + 1, an improved policy π _t ₊₁ is generated by acting greedily (exploitation) with respect to $Q^{π}_{t}$ . In this framework, the policy is also referred to as the actor and the Q-function the critic. The goal of PI is to learn an optimal Q-function Q* such that it satisfies the condition

Q^{*} (s, a) > Q^{*} (s, a) \forall s \in S, a \in A, \forall π

Consequently, an optimal policy π ^∗ can be automatically derived from Q* as

π^{*} (s) = {argmax}_{a} Q^{*} (s, a)

For a finite-dimensional MDP, the Bellman equation is employed to analytically optimize the Q-function. However, for most real-world applications, including the problem-at-hand, prior knowledge of environment dynamics is rendered unknown due to stochasticity. The most powerful approach in this scenario is temporal difference (TD)-based PI which is model free and learns incrementally from the data.⁴⁰ There are two classical TD learning algorithms applied in the context of control, that is, Q-learning (off-policy TD learning) and SARSA (on-policy TD learning), where the focus of this work lies on the latter approach.

In SARSA, an arbitrarily initialized Q-function is incrementally updated with a tuple $< s_{t}, a_{t}, r_{t + 1}, s_{t + 1}, a_{t + 1} >$ as follows

Q_{t + 1} (s_{t}, a_{t}) = Q_{t} (s_{t}, a_{t}) + α * [r_{t + 1} + γ Q_{t} (s_{t + 1}, a_{t + 1}) - Q_{t} (s_{t}, a_{t})]

where $α \in [0, 1)$ is the learning-rate/step-size parameter; the term between square brackets is the TD error between the current estimate $Q_{t} (s_{t}, a_{t})$ and the updated estimate $r_{t + 1} + γ Q_{t} (s_{t + 1}, a_{t + 1})$ . Equation (4) highlights that every update aims to estimate the Q-function of the policy being followed, that is, on-policy TD learning.

This learning process can be further sped-up by incorporating eligibility traces e ∈ ℝ which update Q not only by the current tuple but rather with all visited state-action pairs that lead to this tuple as it is the causal result of an entire trajectory. This variant of SARSA, more commonly known as SARSA(⋋) with accumulating eligibility traces,⁴⁵ has memory parameters associated with each state-action pair that are initialized to 0 at the beginning of each trajectory and updated as

e_{t} (s, a) = {\begin{matrix} γ ⋋ e_{t - 1} (s, a) + 1 if s = s_{t}, a = a_{t} \\ γ ⋋ e_{t - 1} (s, a) otherwise \end{matrix}

where $⋋ \in [0, 1)$ is the accumulating trace-decay error that controls the degree of how fast a trace is decreased to zero. Then, equation (3) is modified to update Q as follows

Q_{t + 1} (s_{t}, a_{t}) = Q_{t} (s_{t}, a_{t}) + α * [r_{t + 1} + γ Q_{t} (s_{t + 1}, a_{t + 1}) - Q_{t} (s_{t}, a_{t})] * e_{t} (s, a)

Convergence to Q* is dependent upon two factors: (i) Robbins-Monro criteria:⁴⁶ the learning rate α should be optimized in the range $α \in [0, 1]$ to satisfy $\sum_{i = 0}^{\infty} a_{i} = \infty$ and $\sum_{i = 0}^{\infty} a_{i}^{2} < \infty$ ; (ii) exploration:⁴⁷ to ensure that every state-action pair is visited an infinite number of times. The former criteria can be achieved through a manual selection process whereas the latter requires that the updates occur by following a stochastic policy (e.g. Є –greedy, softmax, etc.) that will explore the environment by selecting random actions with certain probability measure. Additionally for SARSA, this exploration should be reduced to 0 as t →∞. Under these conditions, an optimal memoryless stationary policy, that is, π* = π ∀ t can be extracted from the Q-function to reach the target according to equation (3). PI is motivating due to its ability to reach optimality with asymptotic performance.

However, fulfilling criteria (ii) is nontrivial for a continuous domain environment, necessitating techniques to approximate the Q-function, as discussed subsequently.

Parametric linear function approximation

For continuous domains, the Q-function can be estimated either with linear/non-linear function approximation. Theoretical stability and convergence properties favor linear approximators over non-linear approximators.⁴⁸

The choice opted for the parametric linear function approximator is tile coding, which partitions the state space into multiple layers called tilings in the spherical coordinate system such that R = [0 max(R;)]; (ii) ⱷ =[0° 360°]; and (iii) θ = [max(θ) 90°]; where max(R) refers to the length of the manipulator in a fully extended state; ⱷ refers to the azimuth which for omnidirectional bending will always be 360°; and θ refers the contraction capability of the manipulator measured from the z-axis. The complete feature space is then represented by rectangular shaped tilings (refer to Figure 9) and is automatically restricted to the reachable workspace of the manipulator. Each tiling is divided into elemental blocks called a tile that allows for local generalization of distances close in Euclidean space. The resolution is different in each dimension of the state-space, given mathematically as

ϕ_{i} = \frac{l_{i}}{N}

Figure 9.

3-D Cartesian plane is abstracted into 2-layered rectangular tilings in R, ⱷ, and θ dimensions. There are 7 tiles, 8 tiles, and 1 tile with a width of wR, wⱷ, and wθ in each dimension, respectively. The origin of the tilings are offset with respect to each other. The position of the soft manipulator end effector activates one tile per tiling.

where i = 1,…, n and n is the total number of dimensions of the state-space; ϕ represents the resolution of a tile in a given dimension; l represents the width of the tile in given dimension; N represents the total number of tilings. The role of resolution is crucial on the learning performance of the algorithm: A larger resolution implies slower learning but more optimal solutions, whereas smaller tiles imply faster learning but less optimal solutions. The selection of the resolution is mostly done through a manual trial and error process. Tiles act as receptive fields, that is, only one tile per tiling is activated if and only if the given state falls in the region delineated by that tile. The origin of each tiling is offset to avoid singularities at the boundaries of the tiles. Q-function is then simply approximated by a sum of the indexes of these activated tiles as

Q (s, a) = {\sum_{j = 1}^{k} Θ}_{j} (s) w_{j}

where j = 1…k, where k is the total number of tilings; θ : S x A → ℝ^k is a vector of binary basis functions ${[θ_{1} (s, a),\dots, θ_{k} (s, a)]}^{T}$ ; w ∈ ℝ^k is the parameter vector. Equation (8) shows that the computational complexity of tile coding is linearly dependent on the number of tilings. Parameter tuning for tile coding is done offline and kept constant throughout the experimental work. SARSA(⋋) can be combined with linear approximation by using gradient-based updates which will change the parameter vector according to,

w_{t + 1} = w_{t} + α_{t} * [r_{t + 1} + γ θ {(s_{t + 1}, a_{t + 1})}^{T} w_{t} - θ {(s_{t}, a_{t})}^{T} w_{t}] * \frac{\partial}{\partial w_{t}} Q (s_{t}, a_{t}) * e_{t}

Reward structure and policy learning

In order to facilitate learning, we guide the policies through the technique of reward shaping such that the MAS is motivated to select actions that progressively lead towards the goal. (Note: In this work, the optimal policy is learnt from a single starting state towards the goal). This is implemented in an information theoretic manner by defining the reward using a distance measure from the goal as follows: A bounded monotonically increasing reward function is implemented representing unequally spaced concentric spheres centered on the target, where the number of spheres in this work are selected randomly but once created, is applied in a similar manner for any target. The motivation behind this reward structure is to allot the distribution of information throughout the abstracted state-space such that motor commands that result in positions with distances further from the goal will be updated with a higher negative value in comparison to targets closer to the goal. Even though this provides efficient distribution of information, it would still suffer from the curse of dimensionality⁴⁹ as the policy space increases exponentially with the number of agents. Thus, we modularize the learning process that is the policy learns to reach the goal by learning by moving through the state-space in the direction of greater reward. This is achieved by taking advantage of our reward structure such that once it encounters an action that is rewarded with a scalar value higher than previously encountered ones, it will be the first action taken by the system in the next episode onwards.

Hierarchical controller

A schematic overview of the motion controller is depicted in Figure 10. It has a hierarchical architecture such that at the higher level, a trajectory is generated within the reachable workspace of the manipulator using the spherical coordinates. It is sampled at n points $[s_{0}, s_{1}, \dots, s_{n}]$ and follows them in a sequential order. In structured environments, this could be used to generate circular, curved, and so on, trajectories offline. However, in this work, we are motivated by the unpredictability of the showering environment and, hence, aim to test the movement of the module through an asymmetrical trajectory which essentially implies that the task space is sampled at random points in the state-space in a sequential order with the aim of the manipulator to reach these points in the same order. These samples are then fed sequentially (s_i) to the low-level module where it generates a solution to reach the given Cartesian spatial target position using the control framework discussed in section “Control framework”. Once generated, the next sample is requested.

Figure 10.

Schematic view of the hierarchical control architecture for a point-to-point motion controller for soft robotic modules.

Experiments

The implementation of the tile coding software was achieved through typically available software. Before using it, all the necessary parameters required by the algorithm are set as follows (refer to Table 2): (i) reachable workspace: It is defined in spherical coordinates in consistence with the characterization of the module shown in Figure 8, that is, R = [10 cm 30 cm], ⱷ = [0° 360°], θ = [0° 90°]. The upper range of the radius was overextended to compensate for the unseen scenarios, for example, the tear of a cable resulting in a larger range for the workspace. (ii) Resolution: 10, 12, and 10 tiles were initialized in the R, ⱷ, and θ dimensions with a total of 4 tilings offset with respect to one another such that the state space has a total of 4800 tiles. (iii) Learning parameters: The step-size (α) and exploration (Є) are set to 0.16 and 0.05, standard values found in text. The eligibility traces (ƛ) are set to 0.9. (iv) Accuracy: The radius of the target circle was kept at a randomly chosen value of .99 cm. This indicates the maximum accuracy that is acceptable as a solution, the minimum being 0. (v) Action-set: The module is allowed to choose actions from a heuristically initialized discrete action-set that can simultaneously decrease/increase/keep unchanged the length of the cables and pneumatics. The length of the cable is controlled by a servomotor rotation range of [0°, 180°], whereas the length of the chamber is controlled by pressure-regulating valves in the range of [0,1] bar. The cardinality of the action-set implies that the module has to able to learn the required motor command from a total of 9⁶ = 531,441 input action combinations. (vi) Algorithmic settings: The algorithm runs online for a total of 100 episodes with 50 trials per episode and is evaluated on two criteria: (a) mean reaching error and (b) trials needed for convergence.

Table 2.

A summary of applied parameters for the proximal module.

Module length	20.5 cm
Reachable workspace	R	10–30 cm	nR	10	wR	0.02
Reachable workspace	ⱷ	0°–360°	nⱷ	12	wⱷ	0.52
	θ	0°–45.8°	nθ	10	wθ	0.08
State-space parameters	T	4
State-space parameters	α	0.16
	Є	0.1
	⋋	0.9
	ɣ	1
Action set	Pneumatic chambers	[−0.087 −0.0349 −0.0175 0.0175 0.0349 0.087] (bar)
Action set	Servomotors	[−25 −20 −10 −5 0 5 10 20 25]°

Experimental setup

The experimental setup is similar to the one mentioned in section “Kinetic characterization of the distal module”; however, an OptiTrack Motion Capture V120: Trio vision feedback system was employed to provide real-time 3-D positioning data. There are seven retroreflective markers attached to the end effector in a random manner such that four markers need to be detected for the 3-D reconstruction of the tip position. The complete setup with the initial position of the robotic module is shown in Figure 11. The system represents a mapping from an R⁶ motor space to an R³ Cartesian space.

Figure 11.

Experimental setup.

Results

We generated 12 points in an uneven shaped curve starting from the top right end with respect to the module in the resting position moving downward toward the resting position. The higher-level controller then fed each sample sequentially to the low-level controller.

Reaching error

Figure 12 depicts the kinematic solution learnt by the manipulator for each sample. The algorithm was found to converge for all 12 points with an average reaching accuracy of 0.79 cm (± 0.18) as summarized in Table 2. It is worthy to mention that this result for accuracy opens the possibility to apply this module to reach sensitive areas for providing future services of scrubbing and rinsing. Furthermore, the possibility to control both pneumatic and cable actuators simultaneously implies that the control can inherently produce stiffness, which otherwise require tedious kinematic and force mappings. Thus, it facilitates a model-free form of handling internal/external loads in an efficient manner. The authors would also like to mention that this accuracy can be improved by reducing the target region size, provided that the system can also physically reach that desired accuracy. The green line depicts the trajectory the module moves through to go from one point to another. The trajectory from the resting position to the first target point is not shown as it obstructs the view from the other targets.

Figure 12.

The distal module moving point-to-point through an asymmetric 3-D trajectory generated offline and sampled at 12 points. Each frame corresponds to the learnt kinematic solution of the manipulator for a given sample. The green line depicts the trajectory followed by the manipulator from sample to sample.

Convergence time

For a given sample, the algorithm allows the module to test a total of 50,000 actions (i.e. 100 episodes × 50 trials per episode); however, Table 3 demonstrates that the algorithm is able to generate solutions for all the given samples within a couple of hundred of actions apart from targets 1 and 2, which will be discussed in the next paragraph. This is a direct consequence of the reward structure mentioned in the previous section. In a state where the manipulator has experienced actions with both lower and higher scalar reward, it will prefer actions with higher reward. This implies high data efficiency but also the ability to quickly learn optimal solutions high in accuracy. However, as a future research goal, the authors intend to use the hierarchical control in conjunction with a learnt forward model. A forward mapping is a causal unique relation between the motor and task space, and thus, are accurate to learn. The inverse kinematic solutions can then be learnt without executing real-time trajectories, saving both time and hardware resources.

Table 3.

A summary of the obtained results for point-to-point motion control of the module through a sampled asymmetric trajectory.^a

No. of goals reached	12
Desired accuracy	0.99 cm
Average reaching accuracy	0.79 cm ± 0.18 cm
Sample number	1	2	3	4	5	6	7	8	9	10	11	12
Reaching accuracy (cm)	.93	.59	.63	.93	.81	.79	.94	.66	.88	.98	.4	.93
Actions taken per point before convergence	5394	4396	16	1020	787	410	30	423	27	8	230	661

^aNote: The data from this table are color coordinated with the data provided in Figure 13.

Targets 1 and 2 from Table 3 show that a significant amount of actions were required to achieve convergence. This is because the actions taken by the module to reach these two targets positioned the retroreflective markers in such a way that it was challenging to reconstruct the 3-D position for the Opti-track system. As mentioned previously, an occlusion for more than three markers at various places in the state space returns no information to the algorithm which implies that (0,0,0) is returned to the algorithm. This state has been given a reward of −1000. The red plots in Figure 13(b) illustrate the reward accumulation for these two targets, which is significantly much higher as compared to the other samples. Despite this challenge, the algorithm was able to learn a solution as long as the target point was not occluded.

Figure 13.

The reward accumulation for all samples of the asymmetric trajectory.

The quality of the learnt policies

As mentioned in section “Parametric linear function approximation”, the worst performance of this algorithm will be generating a suboptimal policy. In this practical setup, this was found to be true. The algorithm is tested from various starting positions; however, in majority of the cases, the solution was generated from the positions that are closer to the goal rather than further from the goal. One reason for this could be that the resolution of the state-space is not optimized, that is, it should be increased. However, this will result in an exponential increase in computational time, whereas the policy currently learnt very well meets our requirements. Consequently, the authors believe that such an approach should not be precluded from practical applications.

Figure 13 depicts the reward accumulation for the all samples. The same trend is exhibited for all experiments which is that the reward accumulation will increase until convergence where it will receive a reward of 0, hence becoming constant.

Discussion and conclusions

Soft robotics has already demonstrated promising results to revolutionize key sectors such as industry and surgery primarily because this technology guarantees safe human–robot interaction. This article aims to apply the powerful properties of soft robotic systems to design the first system to assist the elderly community in the task of bathing. We proposed an overall modular system; however, in order to reach this ambitious goal, we first needed to design, fabricate, and test the key component module that will perform the main bathing activities, that is the distal module which will come in direct contact with the human.

For the design, a hybrid actuated system was proposed. It comprises of radially arranged bellow-shaped pneumatic chambers and cables. The kinetic characterization of the module pointed out the following functional capabilities of the robotic platform: The novel bellow shape of the fluidic actuators contributes toward elongation and contraction, whereas the fluidic chambers contribute to adapt the tip orientation. The overall advantages provided by the combination of these actuators are (i) reachability to the user workspace: It guarantees to cover larger distances with a reduced number of modules and orient the module with respect to the target region of the user’s body. This was observed through the measured reachable workspace which was 21.5 cm × 17.2 cm × 10 cm in the x–y–z axes, respectively. Consequently, a single module has the capability to match the dimensions of the back region of the human. However, further experimentations are required to test how well this module can be oriented toward this region in a modular form. We also acknowledge that these results are specific to the experimental setup described in section “Experimental setup”. However, this is not considered a source of error as the overall behavior (apart from the asymmetries) is in congruence with theoretical expectations. For future work, a more optimized setup will be prepared by introducing motors instead of the servo-motors to facilitate variable adjustment of the module motion, due to the extrinsic cable-driven actuation. (ii) Stiffness variation: Hybrid actuation provides an antagonistic motion between cables and pneumatic actuators that results in a stiffening which is particularly important to compensate for internal/external loading in the bathing environment.

For the control, a point-to-point motion hierarchical controller based on a multiagent RL was used to control the inverse kinetics of a high-dimensional, nonlinear, hybrid actuated manipulator. We tested the controller to move the manipulator through an asymmetric trajectory generated offline sample at 12 discrete points. We found the algorithm to provide solutions to all points with positional accuracy of 0.79 cm ± 0.18 cm. It is worthy to mention that this accuracy opens the possibility to apply this module to reach sensitive areas for providing future services of scrubbing and rinsing. The accuracy can also be improved by setting the target region to a lower value, of course provided that the system can physically reach that limit. Furthermore, as the positions are being reached with all six input activations, this implies that the control is able to exploit the stiffness properties in a model-free manner while reaching which otherwise is achieved through complex kinematic and force models. The authors also demonstrated the effectiveness of applying this approach despite its ability to generate only suboptimal policies. A limitation of the approach lies in the possible high number of steps needed to reach a single point. This will be addressed through the computation of the forward model on which the proposed method can be applied and then the inverse solution is computed, this can be executed by the robot. Future works will focus on this improvement as well as on the control of the tip orientation which is not considered in this implementation. Also, it is also demonstrated that vision-based systems may introduce additional constraints and hence, it is recommended to find alternative sources of feedback.

Footnotes

Acknowledgements

The authors would like to acknowledge the support by the People Programme (Marie Curie Actions) of the European Union’s Seventh Framework Programme FP7/2007-2013/ under REA grant agreement number 608022; the European Commission through the I-SUPPORT project (#643666); and by the Italian Ministry of Foreign Affairs, General Directorate for the Promotion of the Country System”, Bilateral and Multilateral Scientific and Technological Cooperation Unit, for the support through the Joint Laboratory on Biorobotics Engineering project.

Declaration of conflicting interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The authors received no financial support for the research, authorship, and/or publication of this article.

References

Bock

Georgoulas

Linner

. Towards robotic assisted hygienic services: concept for assisting and automating daily activities in the bathroom. Gerontechnology 2012; 11(2): 362.

Brose

Weber

Salatin

. The role of assistive robotics in the lives of persons with disability. Am J Phys Med Rehabil 2010; 89(6): 509–521.

Katz

Ford

Moskowitz

. Studies of illness in the aged: the index of ADL: a standardized measure of biological and psychosocial function. JAMA 1963; 185(12): 914–919.

Miskelly

. Assistive technology in elderly care. Age Ageing 2001; 30(6): 455–458.

Zollo

Wada

Van der Loos

. Special issue on assistive robotics [from the guest editors]. Robot Autom Mag IEEE 2013; 20(1): 16–19.

http://www.seatedshower.com/

http://www.drivemedical.co.uk/sections/bathroom-toilet-aids

Kirchner

Albiez

Seeland

. Towards Assistive Robotics for Home Rehabilitation. Biodevices 2013; 168–177.

Meng

Lee

. Design issues for assistive robotics for the elderly. Adv Eng Inform 2006; 20(2): 171–186.

10.

Laschi

Mazzolai

Cianchetti

. Soft robotics: technologies and systems pushing the boundaries of robot abilities. Sci Robot 2016; 1: eaah3690.

11.

Kier

Smith

. Tongues, tentacles and trunks: the biomechanics of movement in muscular-hydrostats. Zool J Linn Soc 1985; 83(4): 307–324.

12.

Kim

Laschi

Trimmer

. Soft robotics: a bioinspired evolution in robotics. Trends Biotechnol 2013; 31(5): 287–294.

13.

Trivedi

Rahn

Kier

. Soft robotics: biological inspiration, state of the art, and future research. Appl Bionics Biomech 2008; 5(3): 99–117.

14.

Onal

Rus

. A modular approach to soft robots. In: Biomedical robotics and biomechatronics (BioRob), 2012 4th IEEE RAS & EMBS International Conference, 24 June 2012, pp. 1038–1045. IEEE.

15.

Robinson

Davies

. Continuum robots-a state of the art. In: Robotics and Automation, 1999. Proceedings. 1999 IEEE International Conference on 1999, Vol. 4, pp. 2849–2854. IEEE.

16.

Loeve

Breedveld

Dankelman

. Scopes too flexible and too stiff. IEEE Pulse 2010; 1(3): 26–41.

17.

Immega

Antonelli

. The KSI tentacle manipulator. In: robotics and automation, 1995. proceedings., 1995 IEEE International Conference on, 21 May 1995, Vol. 3, pp. 3149–3154. IEEE.

18.

Tsukagoshi

Kitagawa

Segawa

. Active hose: an artificial elephant’s nose with maneuverability for rescue operation. In: Robotics and automation, 2001. proceedings 2001 ICRA. IEEE international conference on, 2001, Vol. 3. IEEE.

19.

Pritts

Rahn

. Design of an artificial muscle continuum robot. In: Robotics and automation, 2004, Proceedings. ICRA’04. 2004 IEEE International Conference on, 26 April 2004, vol. 5, pp. 4742–4746. IEEE.

20.

McMahan

Jones

Walker

. Design and implementation of a multi-section continuum robot: Air-Octor. In: Intelligent Robots and Systems, 2005. (IROS 2005). 2005 IEEE/RSJ International Conference on, 2 August 2005, pp. 2578–2585. IEEE.

21.

McMahan

Chitrakaran

Csencsits

. Field trials and testing of the OctArm continuum manipulator. In: Proceedings 2006 IEEE International Conference on Robotics and Automation, 2006. ICRA, 2006. IEEE.

22.

Kapadia

Walker

. Task-space control of extensible continuum manipulators. In: 2011 IEEE/RSJ International Conference on Intelligent Robots and Systems.

23.

Braganza

Dawson

Walker

. A neural network controller for continuum robots. IEEE Trans Robot 2007; 23(6): 1270–1277.

24.

Giorelli

Renda

Ferri

. A feed-forward neural network learning the inverse kinetics of a soft cable-driven manipulator moving in three-dimensional space. In: Intelligent Robots and Systems (IROS), 2013 IEEE/RSJ International Conference on, 3 Nov 2013, pp. 5033–5039. IEEE.

25.

Grzesiak

Becker

Verl

. The bionic handling assistant: a success story of additive manufacturing. Assem Autom 2011; 31(4): 329–333.

26.

Rolf

Steil

. Efficient exploratory learning of inverse kinematics on a bionic elephant trunk. Neural Networks and Learning Systems. IEEE Trans 2014; 25(6): 1147–1160.

27.

Cheng

Lobovsky

Keating

. Design and analysis of a robust, low-cost, highly articulated manipulator enabled by jamming of granular media. In: Robotics and Automation (ICRA), 2012 IEEE International Conference on, 2012. IEEE.

28.

Stilli

Helge

Althoefer

. Shrinkable, stiffness-controllable soft manipulator based on a bio-inspired antagonistic actuation principle. In: 2014 IEEE/RSJ International Conference on Intelligent Robots and Systems, 2014. IEEE.

29.

Ranzani

Cianchetti

Gerboni

. A Soft modular manipulator for minimally invasive surgery: design and characterization of a single module.

30.

Shiva

Stilli

Noh

. Tendon-based stiffening for a pneumatically actuated soft manipulator. IEEE Robot Autom Lett 2016; 1(2): 632–637.

31.

Marchese

Rus

. Design, kinematics, and control of a soft spatial fluidic elastomer manipulator. Int J Robot Res 2016; 35(7).

32.

Yang

Zhang

Bao

. Research on novel flexible pneumatic actuator FPA. In: Robotics, Automation and Mechatronics, 2004 IEEE Conference on, 1 December 2004, Vol. 1, pp. 385–389. IEEE.

33.

Kang

Branson

Zheng

. Design, modeling and control of a pneumatically actuated manipulator inspired by biological continuum structures. Bioinspir Biomim 2013; 8(3): 036008.

34.

De Volder

Moers

Reynaerts

. Fabrication and control of miniature mckibben actuators. Sensor Actuat A 2011; 166(1): 111–116.

35.

Manti

Pratesi

Falotico

. Soft Assistive Robot for Personal Care of Elderly People. In: 6th IEEE RAS / EMBS International Conference on Biomedical Robotics and Biomechatronics, 2016

36.

Vannucci

Cauli

Falotico

. Adaptive visual pursuit involving eye-head coordination and prediction of the target motion. In: IEEE-RAS International Conference on Humanoid Robots, 2014, pp. 541–546

37.

Vannucci

Falotico

Di Lecce

. Integrating feedback and predictive control in a bio-inspired model of visual pursuit implemented on a humanoid robot. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) 2015; 9222: 256–267

38.

Ansari

Falotico

Mollard

. A Multi-agent reinforcement learning approach for inverse kinematics of high dimensional manipulators with precision positioning. In: 6th IEEE RAS / EMBS International Conference on Biomedical Robotics and Biomechatronics, 2016.

39.

Wooldridge

. An introduction to multiagent systems. John Wiley & Sons, 2009.

40.

Sutton

Barto

. Reinforcement learning: An introduction. Cambridge: MIT Press, 1998.

41.

Claus

Boutilier

. The dynamics of reinforcement learning in cooperative multiagent systems. In: AAAI/IAAI, 26 7 1998, pp. 746–752.

42.

Sandholm

Robert

. Multiagent reinforcement learning in the iterated prisoner’s dilemma. Biosystems 1996; 37(1): 147–166.

43.

Watkins

. Learning from delayed rewards. University of Cambridge, 1989.

44.

Howard

. Dynamic programming and Markov processes, 1960.

45.

Rummery

Mahesan

. On-line Q-learning using connectionist systems, University of Cambridge, Department of Engineering, 1994.

46.

Robbins

Sutton

. A stochastic approximation method. Ann Math Stat 1951: 400–407.

47.

Singh

Jaakkola

Littman

. Convergence results for single-step on-policy reinforcement-learning algorithms. Mach Learn 2000; 38(3): 287–308.

48.

Singh

Jaakkola

Littman

. Convergence results for single-step on-policy reinforcement-learning algorithms. Mach Learn 2000; 38(3): 287–308.

49.

Bellman

. Dynamic programming. Princeton UniversityPress, BellmanDynamic programming, 1957: 151.

Towards the development of a soft manipulator as an assistive robot for personal care of elderly people

Abstract

Keywords

Introduction

Soft robotic manipulators

Challenges in design

Challenges in control

Design

Design and manufacturing of the distal module

Design of the McKibben actuator

Kinetic characterization of the distal module

Experimental setup

Elongation/contraction

Bending and stiffening

Reachable workspace

Control framework

Reinforcement learning preliminaries

Parametric linear function approximation

Reward structure and policy learning

Hierarchical controller

Experiments

Experimental setup

Results

Reaching error

Convergence time

The quality of the learnt policies

Discussion and conclusions

Footnotes

Acknowledgements

Declaration of conflicting interests

Funding

References