Online human-like redundancy optimization for tele-operated anthropomorphic manipulators

Abstract

Robot human-like behavior can enhance the performance of human–robot cooperation with prominently improved natural interaction. This also holds for redundant robots with an anthropomorphic kinematics. In this article, we translated human ability of managing redundancy to control a seven degrees of freedom anthropomorphic robot arm (LWR4+, KUKA, Germany) during tele-operated tasks. We implemented a nonlinear regression method—based on neural networks—between the human arm elbow swivel angle and the hand target pose to achieve an anthropomorphic arm posture during tele-operation tasks. The method was assessed in simulation and experiments were performed with virtual reality tracking tasks in a lab environment. The results showed that the robot achieves a human-like arm posture during tele-operation, and the subjects prefer to work with the biologically inspired robot. The proposed method can be applied in control of anthropomorphic robot manipulators for tele-operated collaborative tasks, such as in factories or in operating rooms.

Keywords

Human-like motion neural network robot redundancy optimization tele-operation vision feedback

Introduction

During the past decades, the demand for robot assistance in human–robot sharing workspace has been more and more popular in various fields, from industry (e.g. manufacturing and assembly) to service robotics (e.g. robot waiter), medical sector (e.g. robotic nurse), where arms need to share the same workspace with humans in the operating room.^1,2 In these cases, robots are expected to behave in a social, predictable manner.³

To ease the cooperation in the shared workspace for human and robotic collaborators, several studies have focused on human–robot cooperation analysis. No matter for robot appearance or behavior, human-likelihood attracts more attention for empowering human efficiency and stamina, relieving the human participants from tedious, hard, or even dangerous tasks.³ With humanoid appearances, redundant anthropomorphic robot arms show their superiority in human–robot interaction,⁴ for example, LWR4+ (KUKA, Augsburg, Germany), Justin robot (Institute of Robotics and Mechatronics, Wessling, Germany), and YuMi (ABB, Zurich, Switzerland). Humanoid robots with anthropomorphic structure are more appropriate when the task demands people working in the robots’ working space.⁵

Several studies have been conducted to improve the assistance quality of humanoid robot manipulators, which also played a fundamental role in human–robot collaborative tasks. A second-order polynomial⁶ was utilized to represent the relation between the wrist pose (referred to the ground) and the arm angle for realizing a human-like control of a redundant robot. The “tutor-tutee hand-in-hand” teaching scheme integrated the tutor’s motor functionalities into the robot’s control architecture, transferring human to human skill.⁷ Natural redundancy resolution in dual-arm manipulation was achieved using humanoid manipulators.⁸ Human impedance transfer skills⁹ were adopted for the control of a dual-arm exoskeleton robot.¹⁰ Human-like motion and compliance properties were achieved and assessed for robot-environment interactions.¹¹ An adaptive hands-on control was realized for reaching and targeting tasks in surgery with redundant robot.^12,13 A trajectory planning system for anthropomorphic robot arm was designed based on an artificial neural network (ANN) architecture¹⁴ trained on human actions.¹⁵ A wrist-elbow-in-line method was derived to map key positions of human demonstrations to the real robot for obtaining a human-like inverse kinematics solution.¹⁶

Kinematic redundancy can also be exploited for achieving human-like behavior.¹⁷ Many optimality criteria with different objective functions, including jerk minimization, joints constraints, and joints torque optimization, have been introduced to model natural trajectories and achieve human-like motion.^18
–20 However, based on the kinematic level, the joints distribution and structure of human arm is different from the real robot arm. In this way, to constrain the joint angles to achieve human-like motion cannot be viewed as an universal method for the redundant robot arm with different kinematic design. The pioneer works have investigated the relation between the swivel angle and the hand pose, synthesizing a human-like behavior for a dual-arm robotic anthropomorphic manipulator (FRIDA, ABB, Sweden).²¹ The kinematic redundancy mapping strategy is thought to facilitate social acceptance, by reducing the stress in human–robot coexistence.²² This method can be chosen as an universal method for redundant robot, because it is independent from joint configuration.

For tele-operated manipulators in industrial or medical area, the end-effector should be accurately controlled by remote master device. In this article, we investigate the redundancy problem resolution by utilizing a neural network (NN) to obtain the nonlinear relation between the 6-D task pose and the 1-D swivel angle of the elbow, based on actual human motion data studies. Then, we transferred the static mapping model for continuous movement control while handling tele-operation tracking task. Our approach aims to achieve human-like motion without violating the joint position and velocity limitations. The particular scenario considered in this article is to implement online human-like redundancy management module able to imitate the human motion, easing the cooperation in a shared human–robot environment, for example, operating room, or an industrial workspace. At the same time, manipulability performance of robot manipulator for online tele-operation was also analyzed after optimization.

Methods

To develop a biologically inspired tele-operated surgeon robot arm in Figure 1, the elbow swivel angle is adopted to achieve human-like arm pose. Human arm motion data (skeleton trajectory) are acquired with an external sensor and then analyzed and integrated in the human arm model in Figure 2.²³ The human arm can be simply modeled as a rigid kinematic chain connected by three joints (shoulder, elbow, wrist) with seven joints ( $q_{1}, q_{2}, q_{3}, q_{4}, q_{5}, q_{6}, q_{7}$ ). The coordinates ( $d_{1}, d_{3}, d_{5}, d_{7}$ ) are the corresponding link lengths which connect torso, shoulder, elbow, wrist, and hand. The shoulder structure including three joints ( $q_{1}, q_{2}, q_{3}$ ) determines the motion of upper arm. Respectively, q₁ determines its forward and backward motion, while q₂ determines its downward and upward motion, and q₃ determines its rotation. The elbow structure with one joint q₄ determines the extension and flexion of forearm. The wrist structure with three joints ( $q_{5}, q_{6}, q_{7}$ ) determines the motion of hand, where q₅ represents rotation of forearm, q₆ represents the extension and flexion of hand, and q₇ represents the rotation of hand. The corresponding Denavit–Hartenberg (DH) parameters are defined in Table 1.²⁴ Joint angles and swivel angle of human arm are calculated online based on the geometry structure of the arm in Figure 3. S_L and S_R are the left shoulder and right shoulder, M is the midpoint of shoulders, T represents the position of torso, E represents the elbow, W represents the right wrist, and H represents the right hand. ψ is the swivel angle between the reference plane and arm plane, $θ_{j}$ is the angle of joint $q_{j}$ , $j = 1, 2, \dots 7$ . The joint 3-D positions are acquired from the Kinect (Microsoft, Washington, USA) with skeleton tracking. The corresponding joint angles are calculated based on the geometry relation. $θ_{1}$ determines the angle between the reference plane and arm plane. $θ_{2}$ determines the angle between vector $S_{R} E$ and $S_{R} T$ , $θ_{3}$ determines rotation angle of the upper arm, $θ_{4}$ determines the angle between link $S_{R} E$ and EW, $θ_{5}$ determines the rotation angle of lower arm, and $θ_{6}$ determines the angle between EW and WH. Since the joint angle $θ_{7}$ cannot be calculated from the sensor data from Kinect,²⁵ additional sensor is required to measure it. In this article, to simplify the analysis, we take $θ_{7} \equiv 0, d_{7} \equiv 0$ . Hand poses of human arm are calculated with joint angles based on DH parameters of the link structure. According to human arm DH parameter (Table 1), the forward kinematic problem between target pose $^{0} T_{7}$ and joints coordinates ( $θ_{j}, j = 1, 2, \dots 7$ ) can be solved with coordination transformation matrix²⁶ $^{j - 1} T_{j}$ from joint $j - 1$ to joint j, where

​^{j - 1} T_{j} = [\begin{matrix} cos θ_{j} & - cos α_{j} sin θ_{j} & sin α_{j} sin θ_{j} & 0 \\ sin θ_{j} & cos α_{j} cos θ_{j} & - sin α_{j} cos θ_{j} & 0 \\ 0 & sin α_{j} & cos α_{j} & d_{j} \\ 0 & 0 & 0 & 1 \end{matrix}]

Figure 1.

Robot human-like arm pose and swivel angle. S is the shoulder position, E is the elbow position, W is the wrist position, the vertical blue plane passing through SW represents reference plane, the red plane represents the arm plane passing through S, E, and W, ψ is the swivel angle between the reference plane and the arm plane.

Figure 2.

Human arm model: $q_{1}, q_{2}, q_{3}, q_{4}, q_{5}, q_{6}, q_{7}$ represent joint degrees and $d_{1}, d_{3}, d_{5}, d_{7}$ represent limb lengths.

Table 1.

DH parameters of the human arm model.

Joint i	$θ_{i}$	$α_{i}$ (rad)	$d_{i}$ (m)
1	$θ_{1}$	$- π / 2$	d ₁
2	$θ_{2}$	$- π / 2$	0
3	$θ_{3}$	$- π / 2$	d ₃
4	$θ_{4}$	$π / 2$	0
5	$θ_{5}$	$π / 2$	d ₅
6	$θ_{6}$	$π / 2$	0
7	$θ_{7}$	$π / 2$	d ₇

DH: Denavit–Hartenberg.

Figure 3.

Geometry of arm model: $S_{R}, S_{L}, T, E, W, H$ represent the 3-D positions of human joints.

Human movement acquisition and processing

With different subjects, the arm length is different. So the DH parameters of model should be also variable. This article focuses on how to transfer the human redundancy optimization ability to tele-operated robot arm. To ignore the influence of different limb length between separate subjects and simplify the model analysis, only one subject is involved in the data collection to acquire arm joints’ angles and hand poses, a human arm motion acquisition and analysis software was implemented with MATLAB 2017a. The software included the 7-degree-of-freedom human arm model in Figure 2 for online visualization of the arm movement. The left picture shows human arm model based on the DH parameters and the right picture shows the skeleton viewer. The human arm model can perform the same motion in the right figure. The hand poses ( $x, y, z, θ_{x}, θ_{y}, θ_{z}$ ) and corresponding arm swivel angles (ψ) were saved during the acquisition, where ( $x, y, z$ ) are the Cartesian position and ( $θ_{x}, θ_{y}, θ_{z}$ ) are corresponding Euler angles. Swivel angles and human hand target pose calculated online were acquired and saved with a rate of about 30 Hz.

The subject was instructed to perform natural reaching motions in the specified cubic task space (surgeons hand workspace) in Figure 4. Data acquisition and analysis were carried out in accordance with the recommendations of our institution with written informed consent from the subject in accordance with the Declaration of Helsinki. The data in Figure 4 were captured and saved with the optical acquisition sensor (Microsoft Kinect V2) placed in front of the subject.

Figure 4.

Performed hand motion is limited in a constrained workspace defined in the front of the body: as it is shown in the picture, the distance from body to constrained workspace is $0.2 m$ , the size of constrained cube workspace is $0.21 \times 0.297 \times 0.18 m^{3}$ , its height from the ground is around $0.69 m$ , and its distance to ground is $0.69 m$ .

NNs architecture and training

Since the regression model between the swivel angle and the target pose of the end-effector has been introduced and solved in nonlinear description,^22,23 a feedforward back-propagation ANN with one hidden layer was implemented to train the regression mapping function in this article. It is known for approximating any function, regardless of its linearity. The nonlinear regression function for mapping the 6-D task pose of human arm model in Figure 2 (including Cartesian positions x,y,z and Euler angles $θ_{x}$ , $θ_{y}$ , $θ_{z}$ ) to its elbow swivel angle (ψ) can be defined as

ψ = f (x, y, z, θ_{x}, θ_{y}, θ_{z})

The number of neurons of the hidden layer was determined by assessing the performances of the regression network. By the comparison tests of motion data, two-layer NN are enough for prediction of the regression function. The final NN architecture including one input layer with 6 inputs (x, y, z, $θ_{x}$ , $θ_{y}$ , $θ_{z}$ ), one hidden layer with 12 neurons and output layer with one output (ψ), updating weight and bias values by Levenberg–Marquardt optimization.²⁷ Levenberg–Marquardt algorithm is the most widely used nonlinear least squares algorithm to calculate the maximum or minimum gradient. It not only has the local convergence of the Gauss–Newton method to minimize those functions, which are the sum of squares of other nonlinear functions, but also has a gradient descent method of global characteristics to look for a new search direction.²⁸ It is adopted in this article for the training of the NN and the performance index is set as the mean square error. The final NN model²⁹ can be written as

Y = W_{2} \cdot (\frac{1}{1 + e^{- (B_{1} + W_{1} X)}}) + B_{2}

Where $X = [x, y, z, θ_{x}, θ_{y}, θ_{z}]$ is the input matrix, $B_{1} = [b_{1}; b_{2}; \dots; b_{12}]$ is the bias matrix of the first layer, $B_{2} = b_{22} \in R$ is the bias of the output layer, and $W_{1} \in R^{12 \times 6}$ and $W_{2} \in R^{1 \times 12}$ are the corresponding weight matrix. The initial condition of the weights and bias is initialized to small random number. In this article, parameters move in the opposite direction of the error to reduce the mean square error to get minimum value. The updating law to determine the weight matrix adopted the increment way.²⁸

The training set was used to update the neurons weights with the predefined number of iterations. When the NN converged to its final configuration, the testing set was used to assess its actual ability to predict human-like swivel angle based on target pose. Two testing data sequences (around 5 min) were given as input for the trained NN.

Robot kinematic control and redundancy resolution

To test the human-like NN model and its feasibility on the control of real robot, a simulation was performed. In Figure 5, O is the intersection of OE and SW. The swivel angle is defined by the angle between the reference plane (BSW) and the actual arm plane (SEW). The swivel angle can be calculated by

ψ = sgn (ψ) arccos (\frac{(\vec{B S} \times \vec{S W}) (\vec{S E} \times \vec{E W})}{| | \vec{B S} \times \vec{S W} | | | | \vec{S E} \times \vec{E W} | |})

where $sgn (ψ) = sgn (\vec{B S} \times \vec{S E} \cdot \vec{S W})$ , $\vec{B S}$ is the vector from the base to the shoulder, $\vec{S} E$ is the vector from the shoulder to the elbow, $\vec{S} W$ is the vector from the shoulder to the wrist, and $\vec{E} W$ is the vector from the elbow to the wrist. Its ideal swivel range is defined as $[- π, π]$ including the green part (feasible areas $[ψ_{min}, ψ_{max}]$ ) and the red part (blocked areas $[- π, ψ_{min})$ and $(ψ_{max}, π]$ . To position the robot in the same workspace as the surgeon arm, the base of the robot was rotated with an angle of 60° around the x-axis. To reduce the influence of difference in link lengths between human arm model and robot arm ([0.31, 0.4, 0.39, 0.22] in meters), the target poses of the robot arm are scaled to 0.5 in Cartesian dimensions for NN prediction. The inverse kinematic is realized with an extension of the analytical inverse kinematic computation algorithm of our previous work.³⁰ Considering joint limitations of the robot arm, the feasible redundancy area in Figure 5 of the robot arm was calculated based on target pose.³¹ The joint limitations from joint q₁ to joint q₇ of LWR4+ robot (KUKA, Augsburg, Germany) are [±170, ±120, ±170, ±120, ±170, ±120, ±170] in degrees.

Figure 5.

Redundancy swivel solution.

For robot tele-operation control, continuous trajectory interpolation should be introduced for online redundancy optimization, avoiding sudden jump in joint space. As it is shown in Figure 6, the trained NN model cannot be applied on the real robot control directly, because its results sometimes are not feasible for joint limitations during the continuous movement. As above, to make the predicted swivel angle from NN feasible for real robot control, the desired swivel angle must be constrained in continuous feasible area. As it can be seen in Figure 7, the redundant robot can reach an unique desired target pose with infinite joint configurations. Firstly, the target pose of end-effector can be acquired from remote master device. It can achieve the desired target pose with infinite swivel angle configuration for its redundancy. However, the swivel range is not always achievable due to kinematic limitations. Only the human-like swivel angle in feasible area can be mapped to feasible joints configurations, achieving human-like arm pose. But for online tele-operation, when there is no feasible human-like swivel solution, the optimization method should achieve the desired target pose with the feasible swivel area. Hence, for online tele-operation, the swivel configuration from NN will be optimized based on the kinematic limitations, constrained to the feasible area. Then, the inverse kinematics will be adopted to get unique joints’ configuration.

Figure 6.

Redundancy optimization of continuous movement: the green areas represent the possible allowed values for the swivel angle during a reaching movement, the white area is blocked area which is not reachable, and the red line is human-like elbow swivel trajectory from NN. NN: neural network.

Figure 7.

Mapping from task space to robot arm pose. The grey areas are feasible configuration for robot without violating kinematic limitations; the blue areas are human-like configuration for robot pose.

In Figure 6, the feasible area of real robot for fixed target pose can be calculated as [ ${\underline{ψ}}_{t}$ , ${\bar{ψ}}_{t}$ ], where ${\underline{ψ}}_{t}$ is the lower boundary and ${\bar{ψ}}_{t}$ is the upper boundary. Before we implement human-like control, we adopted midpoint of feasible interval as one simple method to avoid joint limitations

ψ_{m d} ​_{t} = \frac{{\underline{ψ}}_{t} + {\bar{ψ}}_{t}}{2}

In this way, the desired swivel angle can be constrained in one continuous feasible area. In human-like redundancy resolution, the optimal swivel command $ψ_{h d_{t}}$ is computed as

ψ_{h d_{t}} = k_{1} {\tilde{ψ}}_{t} + k_{2} \frac{{({\bar{ψ}}_{t} - {\underline{ψ}}_{t})}^{2} ({\hat{ψ}}_{t} - {\tilde{ψ}}_{t})}{d}

where $d = 4 π^{2} [{rad}^{2}]$ constrains the constant $\frac{{({\bar{ψ}}_{t} - {\underline{ψ}}_{t})}^{2}}{d} \in (0, 1)$ , ${\hat{ψ}}_{t}$ is the human-like swivel angle predicted from trained NN, ${\tilde{ψ}}_{t}$ is the current swivel angle of the robot arm, $k_{1} > 0$ and $k_{2} > 0$ are constant filtering coefficients, $k_{1} + k_{2} = 1$ and $k_{2} < k_{1}$ to avoid big jump in swivel angle during the motion, the values are adjusted by trial. At the same time, its angle velocity is proportional to the feasible distance $({\bar{ψ}}_{t} - {\underline{ψ}}_{t})$ , which can ensure fast tracking in wide feasible area and avoid fluctuation in narrow feasible area. In this article, $k_{1} = 0.9$ and $k_{2} = 0.1$ . Then the swivel command $ψ_{t}$ must be secured within the feasible boundary $({\underline{ψ}}_{t}, {\bar{ψ}}_{t})$

ψ_{d t} = {\begin{array}{l} ψ_{h d_{t}} & i f ({\underline{ψ}}_{t} < {\hat{ψ}}_{t} < {\bar{ψ}}_{i}) \\ {\underline{ψ}}_{t} + ρ & i f ({\hat{ψ}}_{t} < {\underline{ψ}}_{i}) \\ {\bar{ψ}}_{t} - ρ & i f ({\hat{ψ}}_{t} > {\bar{ψ}}_{i}) \end{array}

where ρ is a small positive constant to avoid reaching the boundary, it is set $ρ = 0.05 [rad]$ in the experiments. Then the inverse kinematic function³¹ was adopted to calculate the joints configuration with input of target pose and swivel angle. To avoid joint speed limitations, the traditional trapezoidal velocity profile is implemented.

System development and architecture

The tele-operated system in Figure 8 includes human motion data analysis and redundant robot control. This robot is kinematic-controlled with fast research interface (FRI) which provides direct low-level real-time access to the KRC (KUKA robot controller) at high rates of up to 1 kHz,³² allowing to implement accurate control strategies. To realize the real-time tele-operation control, the architecture was developed with OROCOS (Open Robotic Control Software, http://www.orocos.org/) application in Ubuntu 12.04 with a real-time Xenomai-patched Linux kernel and robot operating system (ROS, http://www.ros.org/) kinetic in Ubuntu 16.04. The ROS and OROCOS system architecture can be seen in Figure 9. A 720p webcam (30 fps) is adopted for online visual feedback of task execution, providing the resolution of 1280 × 720 pixels in 8-bit per channel, delivering the three RGB components of the images. To ensure the control performance without affection from vision node, the ROS vision node and the control loop were executed on separate computers with UDP communication between each other: the control loop was developed using C++ on a desktop computer with 3.2 GHz Core i7 processor (Intel Corp) and the vision ROS node was developed using OpenCV on a laptop computer with Core i5 processor for 2-D display.³³ The master device working rate was around 1 kHz.

Figure 8.

Flowchart of biologically inspired tele-operated surgeon system.

Figure 9.

ROS and OROCOS architecture. ROS: robot operating system; OROCOS: Open Robotic Control Software.

Finally, the vision node displayed the camera images, messages, reference path, and draw paths on the display and estimated the pose of the camera with respect to the task frame by detecting a set of ArUco markers³³ attached to the task phantom. To make the tracking task safe for tele-operation, the designed curves were drawn with OpenCV in the vision interface, not on the real panel. The experimental setup in Figure 10 comprised a ROS camera vision node interface. Before tele-operation, a calibration procedure was conducted to get the transformation matrix from camera coordinate to robot end-effector frame. The tele-operation scheme implements 6-D pose control with a master device (Sigma 7, Force Dimension, Switzerland).^34,35 During the real-time tele-operation control, data such as the tools actual pose were sent through UDP communication to the laptop at around 30 Hz for drawing the actual trajectory curve with OpenCV on the vision interface.

Figure 10.

The experimental setup: one camera, one master device, and one redundant robot for tele-operation are shown.

Experimental protocol and evaluation

Experiment 1

Experimental protocol

A subject (male, right-arm, 23 years old, 170 cm tall) was instructed for human movement acquisition. His arm link lengths in Figure 11 are measured as [0.2442 m, 0.2008 m, 0.0537 m] with recorded 3-D data from Kinect. Four groups of training data were collected:

six movements on the horizontal plane keeping the fingers parallel to the Z-axis (four on the perimeter and two on the diagonals); for example, A-B in Figure 4;

six movements on the horizontal plane rotating the wrist from an extended position to a flexed position (four on the perimeter and two on the diagonals); for example, C-D;

six movements from the bottom corners to the top corners keeping the fingers parallel to the Z-axis (four on the lateral planes and two on the diagonal planes); for example, E-F; and

six movements from the top corners to the bottom corners keeping the fingers parallel to the Z-axis (four on the lateral planes and two on the diagonal planes); for example, E-F.

Figure 11.

Data acquisition and analysis interface.

Two other groups of data of the same subject involved in data acquisition were collected for NN performance evaluation with arbitrary long movements (around 5 min) in specified space. To train the regression model, the total acquired dataset was divided as:

– training set (24 trajectories); and

– testing set (2 arbitrary movements trajectories).

All movements for training (24 trajectories) were performed twice in natural manner with a speed around 0.05 m/s and repeated twice. Forty six data sets remained, as two were too noisy to be used. Before analysis on the data, a moving median filter (with windows length of six) and a moving average filter (with windows length eight) were applied to the data to smoothen the trajectories and reduce measurement noise.

A NN model is trained to map the hand pose to the arm swivel angle, explicating human arm redundancy resolution method. For performance evaluation of the trained NN, prediction mean error of swivel angle was calculated, and the root mean square error (RMSE) for prediction of human swivel angle was calculated

λ = \sqrt{\frac{{\sum_{i = 1}^{n} ({\hat{ψ}}_{i} - {\tilde{ψ}}_{i})}^{2}}{n}}

where ${\hat{ψ}}_{i}$ is the ith predicted swivel angle, ${\tilde{ψ}}_{i}$ is the ith actual swivel angle, and n is the total step number. The RMSE of the error is calculated between NN-generated swivel trajectories and the recorded actual human swivel trajectories to see if the trained network was comparable with human motion variability. The updating weight $W_{1}$ is shown in Figure 12. It shows the weighting of the NN coverage with the increased data sets. The regression training performance is shown in Figure 13.

Figure 12.

NN regression training performance. NN: neural network.

Figure 13.

Neural network regression training performance.

Results

The RMSE between the proposed NN output and the actual swivel trajectories of two acquired arbitrary hand motion is reported in Table 1.

The trained model is comparable with the subjects’ motion variability on the swivel trajectory.

Experiment 2

Experimental protocol

To evaluate the proposed human-like redundancy optimization method, three different curves in Figure 14 were designed for tele-operation tracking.³⁰ The corresponding actual Cartesian trajectory of end-effector in camera frame would also be drawn on the vision interface. The following two redundancy optimization algorithms were compared with the designed tele-operation experiments:

– mid-point redundancy optimization in equation (3); and

– human-like redundancy optimization in equation (4).

Figure 14.

2-D tele-operation tracking task in vision node. Three curve tasks are designed in above pictures. The green line is desired trajectory, and the blue line is actual trajectory.

A subject (male, 28 years old) was instructed to perform tele-operation control for drawing the same curve on the original task curve using the vision interface shown in Figure 18. The videos of the robot arm movements during the tele-operated control were recorded for further subjective evaluation. For each designed curve task, tele-operation drawing was separately repeated five times with the above two redundancy resolution strategies.

It is known that high manipulability in robotic manipulators could achieve high performance.³⁶ In this article, average manipulability indices χ of three tracking curves during tele-operation were also calculated to give a quantitative comparison of the ability to move in arbitrary directions, including translation and rotation manipulability defined by

χ = \frac{\sum_{i = 1}^{n} \sqrt{‖ J_{i} \cdot J_{i}^{T} ‖}}{n}

where $J_{i}$ is the Jacobian matrix from the base to the end-effector,³⁷ i is the sampled sequence, and n the total step number of the sampled data. The average manipulability indices for three designed curves repeated separately five times.

Results

Swivel performance in Figure 15 and the corresponding joints trajectory in Figure 16 were recorded for comparison. During the tracking tasks, the continuous variation of swivel angle with time can be called as “swivel trajectory.” As can be seen from left picture in Figure 15, the desired swivel trajectory exists in the feasible area but was sudden jumps during the continuous movement. And for the right human-like optimizer, the human-like trajectory sometime is out of the feasible area. Both the actual swivel trajectories are constrained between the upper and lower boundary. Figure 16 shows that the joints trajectory of human-like optimizer is much more smooth than midpoint optimizer, because its swivel trajectory is much more smoother than the midpoint solution.

Figure 15.

Swivel trajectory comparison. Left picture is the result of mid-point optimizer; right picture is human-like optimizer. The dark green line is the upper boundary of the feasible area, the blue line is lower boundary, the red line is desired swivel trajectory, and the green line is actual swivel trajectory.

Figure 16.

Joints trajectory comparison: left picture is the result of midpoint optimizer; right picture is human-like optimizer. The seven curves with different color represent the seven joints trajectory of KUKA robot.

The comparison of manipulation indices in Table 3 between two different inverse kinematic solutions is shown in Table 2. The manipulation indices of human-like redundancy are bigger than the previous midpoint inverse kinematic solution. Thus, the human-like redundancy can achieve better manipulability performance than the midpoint solution.

Table 2.

RMSE between NN-predicted swivel trajectory and actual human swivel trajectory, based on the two testing data set.

Trajectory no.	1	2
RMSE (rad)	0.029	0.023

RMSE: root mean square error; NN: neural network.

Table 3.

Average manipulability index χ on the same tasks.^a

Performance	Human-like (total 15)	Mid-point (total 15)
Task 10	$3.4306 \pm 0.13$	$3.4218 \pm 0.25$
Task 2	$3.4312 \pm 0.22$	$3.2345 \pm 0.21$
Task 3	$3.4632 \pm 0.21$	$3.2106 \pm 0.20$

^a Three corresponding tasks are shown in Figure 14. The manipulability indices are calculated with average value for each tracking task.

Experiment 3

Experimental protocol

Twenty healthy naive subjects (10 females and 10 males, age between 21 and 29 years) were enrolled to evaluate the performance by watching videos without knowing anything about redundancy optimization in Figure 17. The subjects were instructed to compare which video was more human-like; which motion is more familiar; and which is more comfortable to work with in the same room. A noise-canceling headset is used to prevent the subject from being influenced by outside noise.

Figure 17.

Each subject was seated in front of the screen and commented on 15 slides. In each slide, there were one human-like demo and one non-human-like demo, placed with random order.

Figure 18.

Real-time robot control: the above pictures are robot arm poses optimized with two different optimizers; for one same target pose left picture is midpoint optimizer, right picture is with human-like optimizer. One surgeon head picture is placed on the head position and videos are taken in the view parallel to the task plane to reduce influence of robot displacement.

Results

As you can see in Table 4, most of the comments (91%) can tell the difference of human-like or not in motion manner. The comments (84.7%) are familiar with the human-like motion, and 89% of the comments are likely to work with the human-like redundancy-optimized robot arm in the same room.

Table 4.

Subjects’ comments on the slides.^a

Performance	Human-like (total 15)	Mid-point (total 15)
Human-like	273 (91%)	27 (9%)
Familar	254 (84.7%)	46 (15.3%)
Comfortable	267 (89%)	33 (11%)

^aFor performance: “Human-like” represents whether the motion is human-like or not; “familiar” represents whether the motion is familiar for subjects or not; “comfortable” represents whether it’s comfortable for subjects to work with the robot in the same room. For both optimizer, the number of recorded videos are $5 \times 3 = 15$ . And for each video, the number of subjects comment is 20. The percentages represent the proportion of the total number 300.

In conclusion, the proposed NN can achieve human-like arm pose for tele-operation task without violation of kinematic limitations. The subjects are more familiar with human-like motion and feel comfortable to work with the human-like redundancy optimized robot. At the same time, the proposed redundancy resolution method can also improve the manipulability performance for the end-effector. With the trained model, the robotic arm can replicate human kinematics strategies for performing tele-operation tracking task, achieving human-like arm posture during the whole procedure. The method also takes into account the robot kinematic limitations to ensure online motion smoothness and feasibility for tele-operation.

Conclusions

This article developed an online human-like redundancy resolution system for 2-D tele-operation tracking task based on NN. Joint limitations are considered in the redundancy optimization to make the trajectory feasible and smooth. Simulations and tele-operation tracking tasks were conducted with the redundant robot manipulator LWR4+ to verify the performance and feasibility of the proposed algorithm. Results showed that the robot can achieve human-like motion without exceeding the joint limitations. Tele-operator could just focus on the vision tasks, without paying attention to the real robot arm pose. The developed system could make people (e.g. human nurses) more familiar and comfortable to work with surgical robot in the same workspace. The online redundancy optimization model can also be applied in a wide variety of anthropomorphic robot arms with similar structure. However, the task is conducted in a specified task space that depends on the data acquisition, not working for the whole workspace for robot. For other different tasks or workspaces, the human-like model could be required based on human motion data in the new corresponding workspace. The NN model was only trained with data from one subject. In the near future, more subjects and data would be used to improve the model. And the results will be evaluated with surgeons working in the shared workspace with the popularity of surgical robots in the operating room, it can be expected to see more research efforts expended in this field, not only for tele-operation tracking but also for grasping and handing over objects in human-like manner. The future work could also evaluate the practicality of the proposed method with the participation of real nurse in the operating room.

Footnotes

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This project has received funding from the European Unions Horizon 2020 research and innovation program under grant agreement no. H2020-ICT-2016-732515.

ORCID iD

Hang Su

Supplemental material

Supplemental material for this article is available online.

References

Zhao

Hou

. An assistant robot system for sinus surgery. J Med Devices 2016; 10(3): 30923–30925.

Mohareri

Salcudean

. da Vinci^® auxiliary arm as a robotic surgical assistant for semi-autonomous ultrasound guidance during robot-assisted laparoscopic surgery. In Proceedings of the 7th Hamlyn symposium on medical robotics, Imperial College London, UK, pp. 45–46. ISBN: 978-0-9563776-6-1.

Arkin

Moshkina

. Affect in human-robot interaction. Georgia Inst Technol 2014; 1–10.

Coradeschi

Ishiguro

Asada

. Human-inspired robots. IEEE Intell Syst 2006; 21(4): 74–85.

Hinds

Roberts

Jones

. Whose job is it anyway? A study of human-robot interaction in a collaborative task. Hum Comput Interact 2004; 19(1): 151–181.

Kim

Park

. Human-like arm motion generation for humanoid robots using motion capture database. In IEEE/RSJ international conference on intelligent robots and systems, Beijing, China, 9–15 October 2006, pp. 3486–3491. IEEE.

Yang

Liang

Ajoudani

. Development of a robotic teaching interface for human to human skill transfer. In IEEE/RSJ international conference on intelligent robots and systems (IROS), Daejeon, South Korea, 9–14 October 2016, pp. 710–716. IEEE.

Ajoudani

Natural redundancy resolution in dual-arm manipulation using CDS control. In Transferring human impedance regulation skills to robots. New York, USA: Springer, 2016. pp. 61–73.

Buzzi

Ferrigno

Jansma

. On the value of estimating human arm stiffness during virtual teleoperation with robotic manipulators. Front Neurosci 2017; 11: 528.

10.

. Fuzzy approximation-based adaptive backstepping control of an exoskeleton for human upper limbs. IEEE Trans Fuzzy Systems 2015; 23(3): 555–566.

11.

Atawnih

Papageorgiou

Doulgeri

. Reaching for redundant arms with human-like motion and compliance properties. Rob Auton Syst 2014; 62(12): 1731–1741.

12.

Beretta

De Momi

Rodriguez y Baena

. Adaptive hands-on control for reaching and targeting tasks in surgery. Int J Adv Robot Syst 2015; 12(5): 50.

13.

Sandoval

Vieyres

. Safety-enhanced collaborative framework for tele-operated minimally invasive surgery using a 7-DoF torque-controlled robot. Int J Control Autom Syst. Epub ahead of print 30 October 2018. DOI: 10.1007/s12555-017-0486-3.

14.

Hannaford

. A model based recurrent neural network with randomness for efficient control with applications. IEEE T Ind Electron 2018; 1–1.

15.

De Momi

Kranendonk

Valenti

. A neural network-based approach for trajectory planning in robot–human handover tasks. Front Robot AI 2016; 3: 34.

16.

Liu

Chen

Steil

. Analytical inverse kinematics solver for anthropomorphic 7-DOF redundant manipulators with human-like configuration constraints. J Intell Robot Syst 2017; 86(1): 63–79.

17.

Dragan

Srinivasa

. Familiarization to robot motion. In Proceedings of the ACM/IEEE international conference on human-robot interaction, pp. 366–373, 2014. New York, USA: ACM.

18.

Ajoudani

Tsagarakis

Bicchi

. Tele-impedance: towards transferring human impedance regulation skills to robots. In IEEE international conference on robotics and automation (ICRA), pp. 382–388. New York, USA: IEEE.

19.

Kim

Lee

Peternel

. Anticipatory robot assistance for the prevention of human static joint overloading in human–robot collaboration. IEEE Robot Autom Lett 2018; 3(1): 68–75.

20.

Peternel

Tsagarakis

Ajoudani

. A human-robot co-manipulation approach based on human sensorimotor information. IEEE Trans Neural Syst Rehabil Eng 2017; 811–822.

21.

Lamperti

Zanchettin

Rocco

. A redundancy resolution method for an anthropomorphic dual-arm manipulator based on a musculoskeletal criterion. In IEEE/RSJ international conference on intelligent robots and systems (IROS), Hamburg, Germany, 28 September–2 October 2015, pp. 1846–1851. IEEE.

22.

Zanchettin

Bascetta

Rocco

. Acceptability of robotic manipulators in shared working environments through human-like redundancy resolution. Appl Ergon 2013; 44(6): 982–989.

23.

Zanchettin

Rocco

Bascetta

. Kinematic analysis and synthesis of the human arm motion during a manipulation task. In IEEE international conference on robotics and automation (ICRA), Shanghai, China, 9–13 May 2011, pp. 2692–2697. IEEE.

24.

Gaz

Flacco

De Luca

. Identifying the dynamic model used by the KUKA LWR: a reverse engineering approach. In 2014 IEEE international conference on robotics and automation (ICRA). IEEE, pp. 1386–1392.

25.

Alexiadis

Kelly

Daras

. Evaluating a dancer’s performance using kinect-based skeleton tracking. In Proceedings of the 19th ACM international conference on Multimedia, Scottsdale, AZ, USA, 28 November–1 December 2011, pp. 659–662. ACM.

26.

Sciavicco

Siciliano

. Modelling and control of robot manipulators. Berlin, Heidelberg: Springer Science & Business Media, 2012.

27.

Lera

Pinzolas

. Neighborhood based Levenberg–Marquardt algorithm for neural network training. IEEE Trans Neural Netw 2002; 13(5): 1200–1203.

28.

Cui

Xiong

Zheng

. On the application of BP neural network based on Levenberg–Marquardt algorithm in the diagnosis of mental disorders. In 2012 2nd international conference on consumer electronics, communications and networks (CECNet). IEEE, pp. 1940–1943.

29.

Hagan

Demuth

Beale

. Neural network design, vol. 20. Boston: Pws Pub, 1996.

30.

Enayati

Ferrigno

De Momi

. Skill-based human–robot cooperation in tele-operated path tracking. Auton Robot 2018; 42: 997–1009.

31.

Shimizu

Kakuya

Yoon

. Analytical inverse kinematic computation for 7-dof redundant manipulators with joint limits and its application to redundancy resolution. IEEE Trans Robot 2008; 24(5): 1131–1142.

32.

Schreiber

Stemmer

Bischoff

. The fast research interface for the KUKA lightweight robot. In IEEE workshop on innovative robot control architectures for demanding (Research) applications how to modify and enhance commercial controllers (ICRA 2010), Anchorage, Alaska, 2010, pp. 15–21.

33.

Garrido-Jurado

Muñoz-Salinas

Madrid-Cuevas

. Automatic generation and detection of highly reliable fiducial markers under occlusion. Pattern Recognit 2014; 47(6): 2280–2292.

34.

Sandoval

Makhdoomi

. Safety-enhanced human-robot interaction control of redundant robot for teleoperated minimally invasive surgery. In IEEE international conference on robotics and automation (ICRA), Brisbane, Australia, pp. 6611–6616. IEEE.

35.

Sandoval

Vieyres

. Collaborative framework for robot-assisted minimally invasive surgery using a 7-DOF anthropomorphic robot. Rob Auton Syst 2018; 106: 95–106.

36.

Jin

. Manipulability optimization of redundant manipulators using dynamic neural networks. IEEE Trans Ind Electron 2017; 64(6): 4710–4720.

37.

Yoshikawa

. Manipulability of robotic mechanisms. Int J Rob Res 1985; 4(2): 3–9.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB