Toward optimal mapping of human dual-arm motion to humanoid motion for tasks involving contact with the environment

Abstract

In this article, we explore human motion skills in the dual-arm manipulation tasks that include contact with equipment with the final aim to generate human-like humanoid motion. Human motion is analyzed using the optimization approaches starting with the assumption that human motion is optimal. A combination of commonly used optimization criteria in the joint space with the weight coefficients is considered: minimization of kinetic energy, minimization of joint velocities, minimization of the distance between the current and ergonomic positions, and maximization of manipulability. The contribution of each criterion for seven different dual-arm manipulation tasks to provide the most accurate imitation of the human motion is given via suggested inverse optimization approach calculating values of weight coefficients. The effects on actors’ body characteristics and the characteristics of the environment (involved equipment) on the choice of criterion functions are additionally analyzed. The optimal combination of weight coefficients calculated by the inverse optimization approach is used in our inverse kinematics algorithm to transfer human motion skills to the motion of the humanoid robots. The results show that the optimal combination of weight coefficients is able to generate human-like humanoid motions rather than individual one of the considered criterion functions. The recorded human motion and the motion of the humanoid robot ROMEO, obtained with the strategy used by human and defined by our inverse optimal control approach, for the tasks “opening/closing a drawer” are assessed visually and quantitatively.

Keywords

Human motion skills inverse optimal control algorithm criterion functions inverse kinematics algorithm human motion imitation humanoid robots

Introduction

Human motion modeling has been widely studied and explored in the literature with the aim to design and control a humanoid robot inspired by human motion in daily human activities. Human motion can be analyzed in different ways. For instance, a biomechanical perspective is characterized by the need for new information on the characteristics of normal and pathological human movement.¹ In mathematics and robotics, human motion is viewed as a set of differential equations. In order to create a robot with anatomical features close to or resembling those of human beings, humanoid robotics should devote considerable attention to the analysis of human motion characteristics. Humanoid robots have a highly redundant kinematic structure and can be used for the imitation of bioinspired features to model human motion and/or human skills.

There are numerous techniques and strategies to analyze human motion, such as imitation learning and optimization approaches. Imitation learning is based on estimating the characteristics of the demonstrated motion in a task or joint space. Considering the state of the art in the field of human motion imitation, it could be observed that the programming by demonstration is a powerful tool for transfer motion into humanoid robot motion.^2
–4 Since human motion is optimal, we can assume that humans try to minimize the unknown objective function during the manipulation tasks. Accordingly, human motion can be analyzed using optimization approaches with the cost function.⁵

The purpose of this research is to define the objective function optimized in human motion with the aim to transfer human skills to humanoid robots. Although the human model is complex and includes many degrees of freedom (DoFs), our aim is to transfer human skills to a humanoid robot with fewer DoFs. To that end, the scaled kinematic model of the robot ROMEO will be used as a kinematic model of a human body. Human motion tasks which include contact with the environment are analyzed herein. The human motion in task space is partially imposed by the task, while the motion in joint space is free to a certain extent due to redundancy. This type of motion has not been sufficiently explored in the literature. Unlike some other studies where criterion functions are defined in the task space,^6,7 the authors have decided to observe human motion characteristics based on the criterion functions in the joint space and therefore enable an explicit human to humanoid motion transfer. More precisely, we take the basic criterion functions defined in the joint space (minimization of the kinetic energy, minimization of joint velocities, and minimization of the distance between the current position and ergonomic configuration of humans while keeping the arm away from the singularity (maximization manipulability)) which are well-known in the optimization process in robotics. We combine all of these criteria in order to define the combination of criteria which best describes human motion. The optimization process is represented at kinematic level due to the simplicity of the approach and the fact that humanoid robots are often controlled in position, while the torque information is not directly controlled (especially in the case of multiple contacts with the environment as in the present study). The results obtained from the optimization process confirm a priori knowledge about the activation of the upper body joints during the task.⁸ The optimal combination of basic criterion functions, which provides the most accurate imitation of recorded human movement, is based on the analysis of the same motion performed by 15 actors. The effects on the actor’s body characteristics and the characteristics of the environment in the choice of criterion functions are additionally analyzed. The strategy of human motion obtained using the inverse optimal control algorithm is implemented to the robot ROMEO for the purpose of performing the task previously done by a human. Consequently, the desired humanoid motion is generated so to include human motion skills.

The article is structured as follows. “Transfer from a human to a humanoid robot motion” section provides an overview of conversion of the recorded human motion to humanoid robot motion. The inverse optimal control algorithm is presented in “Characterization of human motion using an inverse optimal control approach” section. The following section presents the analysis of seven different types of the dual-arm motions performed by 15 actors using the proposed inverse optimal control algorithm. In “Discussion” section, human skills are transferred to the robot ROMEO for achieving a task done by an actor using an inverse kinematic (IK) algorithm. The experimental results demonstrate that human-like humanoid motion is achieved.

State of the art and challenges in analyzing human motion strategies

This chapter provides an overview of optimization algorithms used for the analysis of human motion as well as the criterion functions frequently used to explain the characteristics of human motion behavior. The final part of this chapter presents the contributions of our research to the field.

Human motion as an optimization problem

Prior to initiating the task execution, it should be mentioned that it is in the nature of every human being to perform a motion in the simplest way and with minimum effort. Therefore, we can assume that humans are always minimizing some unknown criteria in order to create motions. Based on the analysis of human motion, it is possible to extract some criteria that are often used in optimization algorithms in robotics, such as joint limit avoidance, maximization of manipulability, minimization of the magnitude of the velocity and force, joint torques and torque change, kinetic energy, muscle effort, jerk, and so on.^9

–12 The choice of the criterion function depends on the way in which the human motion is analyzed. Hence, the analysis of human movement can be done at the kinematic or dynamic level. Tomić et al.¹³ have examined the impact of each criterion, such as minimization of kinetic energy, velocity, and minimization of deviations from the ergonomic position, on the quality of human motion imitation with and without contact. They used the IK algorithm and included the criterion function using the null space of the Jacobian. Several authors based their studies on defining the dynamic criterion for the purpose of human motion analysis. Khatib et al.¹⁴ analyze the activation of joints and muscles during the motion and define a muscular effort minimization criterion in order to obtain optimal human postures using musculoskeletal dynamics. Their research is based on the biomechanics of human motion. The derived minimization criterion is implemented on the control algorithm of the robot. Zheng and Yaman¹⁵ propose an optimal criterion, which includes the minimization of joint torques and associated contact forces, in order to imitate the given reference motions obtained from human motion capture data. They took into consideration the contact between feet and floor, while the contact between hands and equipment was disregarded.

Upon observing the way in which humans perform the task, it is reasonable to think that humans use combinations of different criterion functions instead of a single criterion, as presented in the article above. Park et al.¹⁶ and Albrecht et al.¹⁷ used the sets of parameters (such as minimization of joint jerks, minimization of torque changes, and so on) to produce the combination of criterion functions for human motion analysis. Mombaur et al.^6,7 defined the imitation of the human locomotion as an optimization problem with an objective function defined in the task space. The objective function is a weighted sum of the basic criterion function such as minimization of total time, integrated squares of the three acceleration components, and the integrated squared difference of the body orientation angle and direction toward the goal. The aim of their research was to produce a universal combination of the weighted coefficients for the optimization algorithm that satisfies the imitation of any type of human locomotion. Billard et al.¹⁸ extend the pseudoinverse optimization method for solving the IKs in order to determine the optimal imitation strategy which best satisfies the constraints of the given task. They defined the objective function as a weighted sum of the basic criterion functions defined into the Cartesian and joint spaces. Their optimization algorithm minimizes the difference between the current and the desired position of the joints and the three-dimensional (3-D) Cartesian position of the hands. They compute the trajectory of robot joints that imitates human motions. The constraints of the robot’s body are taken into account. Likewise, using the joint space, Yang et al.¹⁹ analyzed human motion by combining joint displacement minimization, changes in potential energy, and discomfort basic function in a multiobjective optimization algorithm in order to predict a static posture for the human. The virtual human Santos has been used to evaluate different performance measures and to test the applicability of their optimization algorithm to posture prediction. In each basic function, they proposed the weight coefficient for each joint, taking into account the importance of particular joints for carrying out the task. They applied the optimization algorithm for each basic function separately and compared it with the results obtained for multiobjective optimization.

Unlike some previous studies, our research is based on the analysis of human motion using the inverse optimal control approach with criterion functions defined in the joint space. Given that the human body is a highly redundant system, there are different solutions for performing the same motion in the task space. We seek the combination of the values of the weight coefficients that generates the humanoid motion that is closest to the recorded human motion. This weight will define the criterion optimized by human behavior. Comparing with previous studies, we calculate the values of weight coefficients separately for the different types of the dual-arm motions. In this way, we are able to define a relation between characteristics of the human motion in the joint space and the criterion function which describe them. Furthermore, similar motion patterns in joint space are characterized by a similar criterion function even if the motion in the task space is very different.

Characteristics of human motion joint activation, ergonomy, and manipulability

The task performed by a human is largely determined by muscle activation. Potkonjak et al.⁸ have analyzed motion distribution through the arm joints. The distribution of the motion is connected to the properties of biological systems and the type of hand motion. On the “handwriting” task, humans control their proximal joints for slow movements and positioning, while the movement of the distal joints follows them and controls fine and fast motions. In humans, the high-inertia arm joints (shoulder and elbow) provide the smooth global motion, while the low-inertia hand joints (wrists) perform fast and precise local motions. Based on the biomechanical research, Liu et al.²⁰ analyzed the habits of the human to use some muscles more than others during the motion. Using the nonlinear optimization algorithm, they minimized the energy objective functions, which compute the total amount of torque due to muscle forces and calculate the activity of each muscle during the motion. Yang et al.¹⁹ explain the difference of joint activation via their optimization algorithm, by adding different scalar weight coefficients for each joint. This approach will be used in our optimization algorithm via the weights associated with the minimization of kinetic energies and minimization of joint velocity.

The characteristics of human motion are largely conditioned by the equipment used during the task. If the task does not require interaction with the equipment, a human will choose the most comfortable way to perform the task as explained by Tomić et al.¹³ In other cases, human will adjust their motion to carry out a task and increase comfort as much as possible. Human comfort while performing a task is widely explored in ergonomics, biomechanics, and robotics. Yang et al.²¹ proposed an algorithm based on the combination of IKs, inverse dynamics, and biomechanical information for increasing the comfort level during the motion. The discomfort level of each joint is calculated as a ratio between torques exerted by the joint (calculated using the inverse dynamic) and maximum torque that can be produced by the joint, which is obtained from the ergonomics data. Ma et al.²² combined the conventional posture analysis techniques (proposed in ergonomic analysis) and the fatigue index in the muscles to calculate comfort during the manual handling operations. Yang et al.¹⁹ proposed a discomfort index as an objective way to estimate the most comfortable position of the human body. They defined the ergonomic configuration of the joints, for which the values of the joints are in the middle of their ranges, as the most comfortable position for the human. Since we want to avoid the calculation of the torque, in our research, we will define the discomfort as the difference between the current configuration of joints and the comfortable configuration proposed by Yang et al.¹⁹

The characteristics of the body and the type of the motion affect greatly the feasibility of the task. The ability of the robot to move its end effectors in any direction is presented via the manipulability index.²³ Manipulability index is defined as a ratio between the norm of the motion in joint space and Cartesian space. A joint configuration close to a singularity is characterized by a small value of manipulability. Conversely, the ergonomic configuration of the human body provides human to perform the motion in any direction in the joint and Cartesian spaces. The feasibility of hands’ motion for the dual-arm manipulation tasks performed by a robot is extensively analyzed in the literature.^24,25 Following the same principle, we define one of the criteria in our optimization algorithm which maximized the index of manipulability.

Paper contribution-questions and answers about human motion strategy during the dual-arm manipulation tasks

The present research provides answers to the following questions:

What are the criteria optimized by the motion performed by a healthy human?

Can we represent each human motion as a result of the optimization process and the combination of the basic criterion functions using the inverse optimal control approach?

Do the size of the human, type of contact with the environment, and the distance between the human and the environment have an influence on the criterion function minimized?

Do humans usually choose a specific strategy to generate the same motion with different constraints or the strategies depend on the person performing the motion?

Is the generated strategy obtained from the human motion able to generate a good imitation of the analyzed human motion with a redundant humanoid robot?

In this article, we will show that the inverse optimal control algorithm is able to generate a motion of the scaled model of the humanoid close to the recorded human motion. The weight coefficients of the combination of criterion functions are used as optimization variables. Since we recorded the motion performed by healthy actors, we expect that the combination of the weight coefficients will be the same for the same type of motion and for most actors. Nevertheless, the characteristics of the actors can influence the combination of weight coefficients. We will comment the relationship between the characteristics of the actor motion and the combination of weight coefficients obtained by the optimization process. The final part of this article will show that the IKs algorithm with the combination of the criterion functions, which is calculated during the optimization process, is able to generate a human-like dual-arm motion of the robot ROMEO.

Transfer from a human to a humanoid robot motion

The conversion from human to humanoid motion can be represented in four steps. The first step corresponds to human motion recording. In the second step, a kinematic model of the human is defined based on the humanoid characteristics and the recorded data are processed using the imitation algorithm. The objective is to obtain the motion that most resembles the human motion with the scaled model of the humanoid by taking into account the limited mobility of the humanoid robot versus the human being. In the third step, the inverse optimal control algorithm is used for defining the criterion optimized by human motion. The last step deals with the motion generation of the robot ROMEO using the previously defined criterion. An overview of the conversion from human to humanoid motion in these four steps is given in Figure 1.

Figure 1.

Transfer from a human to a humanoid robot motion—overview.

The skeleton of an adult human consists of 206 bones linked with different types of joints that tend to be flexible. Many humanoid robots have a kinematic arm and leg structure, which consists of seven DoFs per arms and six DoFs per legs with a size of the segments close to the human. A humanoid robot with such characteristics, such as the humanoid robot ROMEO, can produce accurate human motion imitation. In our research, we analyze the dual-arm manipulation in humans using the kinematic model of ROMEO upper body, which consists of 19 DoFs (seven DoFs per arms, two DoFs in the neck, two DoFs in the head, and one DoF in the trunk). We additionally include six DoFs in the trunk, which are not included in the kinematic model of the ROMEO, and emulate the leg and spine motions (three prismatic and three rotation joints (see Figure 2(a)). In this way, we presented the extended kinematic model of the robot ROMEO as a kinematic model of the human. The modified Denavit–Hartenberg convention²⁶ is used to define the extended kinematic model of the robot upper body.

Figure 2.

Kinematic model of the robot ROMEO scaled to the dimension of the human.

The marker-based advanced real-time tracking (ART) motion capture system is used for the human motion recording.²⁷ The sampling frequency for the acquisition data is set to 100 Hz. The dimensions of the robot segments are not exactly the same as the dimensions of the actor’s segments. In order to have joint motions of the robot with all characteristics of human motion, the upper body kinematic model of the robot should be scaled to the dimensions of the human (see Figure 2(b)), obtained from the motion capture system for each actor separately. The dimensions of human segments are calculated by taking the mean Euclidean distance between two adjacent joints and using several data samples taken from the recorded data during the initial configuration. The dimensions of paired segments located on the left and on the right side of the body are assumed to be identical. They are calculated by taking the mean value of the estimated segment dimensions on the right and the left side of each actor.

The imitation algorithm is used to generate the desired motion of scaled model of the robot ROMEO resembling recorded human motion. Our imitation algorithm is based on the use of virtual markers attached to the scaled model of the robot. The configuration of the scaled model of the robot is defined to maximize the matching between the virtual and real marker frames and the position of joints of our model and the model of the actor. Virtual markers are positioned on a scaled model of the humanoid, while the real marker is placed on the human body. The imitation algorithm is formulated as an analytical optimization problem, which calculates the joint configuration of the scaled robot model for each time sample. The trajectories of hands in the Cartesian space, which are used to specify the task, are calculated using the results of the imitation algorithm. The detailed explanation of the imitation algorithm and human motion data recording by means of the ART motion capture system are detailed in the study by Tomić et al.²⁸

Characterization of human motion using an inverse optimal control approach

In this section, an inverse optimal control approach is defined in order to characterize human motion.

The inverse optimal control algorithm

The final goal is to define the optimization criterion used by a human to produce the experimentally recorded motion. Since the studied motion involves at least partially contact with the environment, the criterion is optimized with the constraint that human hands follow a given motion defined by the task. We consider an optimal control problem of the form

[\begin{matrix} min & Φ (q_{t}, {\dot{q}}_{t}) \\ s.t. & \dot{X} = J (q_{t}) {\dot{q}}_{t} \end{matrix}]

where t is a time sample, $q_{t} and {\dot{q}}_{t} \in ℝ^{n}$ are position and velocities of the joints generalized coordinates as a function of t, respectively, $\dot{X}$ velocity vector of the actor’s hands obtained by the imitation algorithm, J(q_t) is the Jacobian matrix of the scaled model of the robot ROMEO that maps joint motion to hand motion, and Φ(⋅) is the objective function that should be minimized. For the objective function Φ(⋅) we make the assumption that it is expressed as a weighted sum of the n base criterion function ϕ_i(⋅) with the corresponding weight coefficient $k_{i} \in ℝ^{+}$

Φ (q_{t}, {\dot{q}}_{t}, k) = \sum_{i = 1}^{n} k_{i} ϕ_{i} (q_{t}, {\dot{q}}_{t})

Based on the literature review of the common criteria that can be minimized using a kinematic model, the set of the base criterion function ϕ_i(⋅) is chosen

1. minimization of the kinetic energy

ϕ_{1} ({\dot{q}}_{t}) = \frac{1}{2} {\dot{q}}_{t}^{T} A {\dot{q}}_{t}

The kinetic energy of the system is a quadratic function of joint velocities ${\dot{q}}_{t}$ where A is the inertia matrix of the dynamic model of the actor, calculated using the Hanavan model of the human body. Using this criterion, the velocity of the shoulder joints (with high inertia) will be reduced with respect to the wrist joints (with small inertia).

2. minimization of the norm of joint velocities

ϕ_{2} ({\dot{q}}_{t}) = \frac{1}{2} {\dot{q}}_{t}^{T} I {\dot{q}}_{t}

The weighted matrix is the identity matrix I.

3. minimization of distance between the current position and the ergonomic configuration of human

ϕ_{3} (q_{t}) = \frac{1}{2} {(q_{t} - q_{e r g o n o m y})}^{T} A (q_{t} - q_{e r g o n o m y})

where q_ergonomy is the ergonomic configuration of the human proposed by Yang et al.¹⁹ The inertia matrix A is chosen in order to include the motion priority of each joint.¹³ In order to represent the objective function Φ(⋅) as a function of ${\dot{q}}_{t}$ , the criterion ϕ₃(q_t) should be adapted as a function of ${\dot{q}}_{t}$ , using the relation between q_t and ${\dot{q}}_{t}$

ϕ_{3} ({\dot{q}}_{t}) = \frac{1}{2} {(q_{t - Δ t} + Δ t {\dot{q}}_{t} - q_{e r g o n o m y})}^{T} A (q_{t - Δ t} + Δ t {\dot{q}}_{t} - q_{e r g o n o m y})

where q_t−Δt is the previous value of the joints generalized coordinate and Δt is the increment of the time calculated according to the frequency for data acquisition.

4. maximization of the manipulability

ϕ_{4} ({\dot{q}}_{t}) = det (J \cdot J^{T})

where J = J(q_t). Since we define the criterion function which should be minimized, the criterion $ϕ_{4} (q_{t})$ is written in the form

ϕ_{4} ({\dot{q}}_{t}) = \frac{1}{2} {({\dot{q}}_{t} - p ω)}^{T} ({\dot{q}}_{t} - p ω)

which is proposed by Zhang et al.²⁹ $ω = \frac{\partial det (J J^{T})}{\partial q_{t}}$ is a vector of manipulability gradient $det (J J^{T})$ and $p \in ℝ^{+}$ is a constant coefficient. The i ^th element of ω is calculated using the equation

ω_{i} = \frac{\partial det (J J^{T})}{\partial q_{t}^{i}} = det (J J^{T}) \cdot trace ({(J J^{T})}^{- 1} (\frac{\partial J}{\partial q_{t}^{i}} J^{T} + J {(\frac{\partial J}{\partial q_{t}^{i}})}^{T}))

Here, trace(⋅) denotes the trace of a matrix argument and $q_{t}^{i}$ is the i ^th element of the vector q_t. This criterion allows that the joint motion tends toward the motion that maximizes manipulability. The term p determines how fast the convergence occurs. The value p $(p = 10^{6})$ is selected to minimize the coefficient of manipulability calculated with $det (J J^{T})$ .

Consequently, the task of determining the best objective function Φ(⋅) is reduced to determining the best weight coefficients $k_{energy}, k_{velocity}, k_{ergonomy}$ , and k_{manipulability} ( $k_{1} = k_{energy}, k_{2} = k_{velocity}, k_{3} = k_{ergonomy}, k_{4} = k_{manipulability}$ . This notation is used to be explicit and easy to read.).

The final equation for objective function $Φ ({\dot{q}}_{t})$ with weight coefficients is

Φ ({\dot{q}}_{t}) = k_{energy} \frac{1}{2} {\dot{q}}_{t}^{T} A {\dot{q}}_{t} + k_{velocity} \frac{1}{2} {\dot{q}}_{t}^{T} I {\dot{q}}_{t} + k_{ergonomy} \frac{1}{2} {(q_{t - Δ t} + Δ t {\dot{q}}_{t} - q_{ergonomy})}^{T} A (q_{t - Δ t} + Δ t {\dot{q}}_{t} - q_{ergonomy}) + k_{manipulability} \frac{1}{2} {({\dot{q}}_{t} - p ω)}^{T} ({\dot{q}}_{t} - p ω)

where $k_{energy}, k_{velocity}, k_{ergonomy}, and k_{manipulability}$ are the weight coefficients which correspond to criterion minimization of kinetic energy, minimization of joint velocities, minimization of the distance between the current position and the ergonomic configuration of human, and maximization of manipulability, respectively. The weights represent the contribution in the percentage of each criterion to the optimal function ( $k_{energy} + k_{velocity} + k_{ergonomy} + k_{manipulability} = 1$ ).

The optimization problem is solved under the constraint $\dot{X} = J (q_{t}) {\dot{q}}_{t}$ given in equation (1) that describes the task to be achieved by the human hand with or without contact with the environment. The task can be integrated into the optimization problem using the Lagrange multiplier mathematical method with a selected set of optimization criteria and constraints

Λ ({\dot{q}}_{t}, λ) = Φ ({\dot{q}}_{t}) + λ (- J (q_{t}) {\dot{q}}_{t} + \dot{X})

The theorem states that the optimal solution is obtained where the gradient of equation (11) with respect to ${\dot{q}}_{t}$ and Lagrange multiplier $λ \in ℝ^{n}$ becomes all zeros. Therefore, the joint velocity for the criterion defined by $k_{energy}, k_{velocity}, k_{ergonomy}, k_{manipulability}$ is

{\dot{q}}_{t} = K (\begin{array}{l} J {(q_{t})}^{T} {(J (q_{t}) K J {(q_{t})}^{T})}^{- 1} \dot{X} + \\ k_{ergonomy} Δ t J {(q_{t})}^{T} {(J (q_{t}) K J {(q_{t})}^{T})}^{- 1} J (q_{t}) K A^{T} (q_{t - Δ t} - q_{ergonomy}) - \\ k_{manipulability} p J {(q_{t})}^{T} {(J (q_{t}) K J {(q_{t})}^{T})}^{- 1} J (q_{t}) K ω - \\ k_{ergonomy} Δ t A^{T} (q_{t - Δ t} - q_{ergonomy}) + k_{manipulability} p ω \end{array})

where $K = {(k_{energy} A^{T} + k_{velocity} I + k_{ergonomy} Δ t^{2} A^{T} + k_{manipulability} I)}^{- 1}$ .

Genetic algorithm for calculation of the weight coefficients

The observed dual-arm model is a redundant system and enables performing the same task differently. Using varied combinations of weight coefficients, we are able to generate different types of motion in the joint space within the same task of the hands in Cartesian space. In such context, it is necessary to define the fitness function, which represents a measure of similarity of generated motion by our inverse optimal control algorithm and the desired movement. Since wrist position error is eliminated by introducing the constraint in the optimization function (11), the fitness function F(⋅) is calculated as an integral of the error between the desired and obtained the position of the shoulder and elbow in Cartesian space using the trapezoidal numerical integration

\begin{array}{l} F (C) = \int_{0}^{t_{end}} E (t, C) d t \approx \frac{t_{end}}{2 N} \sum_{n = 1}^{N} (E (t_{n}, C) - E (t_{n + 1}, C)) \\ E (t_{n}, C) = ‖ {\vec{P}}_{j}^{d} (t_{n}) - {\vec{P}}_{j}^{o} (t_{n}, C) ‖ \\ E (t_{n + 1}, C) = ‖ {\vec{P}}_{j}^{d} (t_{n + 1}) - {\vec{P}}_{j}^{o} (t_{n + 1}, C) ‖ \end{array}

where $C = {k_{energy}, k_{velocity}, k_{ergonomy}, k_{manipulability}}$ is a combination of the weight coefficients, t_n and t_n+1 represent the n ^th and (n + 1)^th time samples, respectively, t_end is the duration of the motion, N is the number of samples during the motion, and E(t_n, C) is the square norm of the error between the vectors of the desired value of the arm joints obtained via the imitation process ${\vec{P}}_{j}^{d} (t_{n}) = [\begin{matrix} P_{RightShoulder}^{d} (t_{n}) & P_{LeftShoulder}^{d} (t_{n}) & P_{RightElbow}^{d} (t_{n}) & P_{LeftElbow}^{d} (t_{n}) \end{matrix}]$ ( $P_{RightShoulder}^{d} (t_{n}), P_{LeftShoulder}^{d} (t_{n}), P_{RightElbow}^{d} (t_{n}), and P_{LeftElbow}^{d} (t_{n})$ are desired positions of right shoulder, left shoulder, right elbow, and left elbow in Cartesian space, respectively) and the position of the arm joints calculated by equation (12) for the C combination of the weight coefficients ${\vec{P}}_{j}^{o} (t_{n}, C) = [\begin{matrix} P_{RightShoulder}^{o} (t_{n}, C) & P_{LeftShoulder}^{o} (t_{n}, C) & P_{RightElbow}^{o} (t_{n}, C) & P_{LeftElbow}^{o} (t_{n}, C) \end{matrix}]$ ( $P_{RightShoulder}^{o} (t_{n}, C), P_{LeftShoulder}^{o} (t_{n}, C), P_{RightElbow}^{o} (t_{n}, C), and P_{LeftElbow}^{o} (t_{n}, C)$ obtained positions of right shoulder, left shoulder, right elbow, and left elbow in Cartesian space, respectively). A small value of the fitness function indicates a good match between the desired and the obtained motion.

Since the fitness function admits many local minima, the algorithm based on the gradient calculation would not always give a global solution. This is why a genetic algorithm, suitable for such context, has been used.³⁰

For the purpose of our research, a genetic algorithm is defined as a multiobjective optimization approach. The genetic algorithm aims to calculate the optimal combination of weight coefficients for which the fitness function defined by equation (13) has a minimum value. Therefore, one combination of weight coefficients $C = {k_{energy}, k_{velocity}, k_{ergonomy}, k_{manipulability}}$ in genetic algorithm notation is one individual. Each individual consists of four chromosomes and represents a unique solution within the solution space. In our case, each chromosome represents one weight coefficient. The value of each weight coefficient is defined in the range $[\begin{matrix} 0 & 1 \end{matrix}]$ and is coded into the genetic algorithm using a real-valued representation of the chromosome. The population of our genetic algorithm is represented by a set of 40 individuals. By testing the genetic algorithm with different numbers of individuals in one population, we concluded that 40 individuals are a good compromise between complexity of our genetic algorithm and its execution time and the quality of optimization. With the aim to cover the complete solution space, a uniform distribution of the individual is used to generate the initial population. The fitness function defined by (13) is evaluated for each individual per generation. The value of the fitness functions is used to determine the possibility of each individual to be selected for the next generation during the selection process. We used the roulette-wheel selection which selects the solutions with the lowest values of the fitness functions but also gives a chance some weaker solutions may survive the selection process. The weaker solution may include some components which may prove useful following the crossover process. The roulette-wheel selection process is selected to prevent convergence toward the local minimum and try to find the global optimal solution. The crossover rate was selected to be equal to 0.7, which means that 70% of the new (children) individuals are made using the crossover operator. The rest children individuals are defined using the mutation operator or are the elite individuals. In our genetic algorithm, two individuals with the lowest values of the fitness function in current generation (elite individuals) are selected and directly included into the next generation with the aim to accelerate the convergence of the genetic algorithm toward an optimal solution. With the aim to make new combinations of the weight coefficients (individuals) which are different from the existing, the mutation operator is used. Mutation provides genetic diversity and enables the genetic algorithm to search a broader space and may bring individuals out of the local minimum and move toward the global one. Our mutation operator adds a unit Gaussian distributed random value to a gene of the chromosome. If it falls outside of lower and upper bounds for the gene, the new gene value is clipped.

The imposed condition for stopping the genetic algorithm is heuristically obtained as the change of the best value of the fitness function F(⋅) is not greater than 10⁻⁶ for the previous 50 generations.

Results

The dual-arm motions are classified as uncoordinated and coordinated with subdivided coordinate motion, as symmetric and asymmetric,³¹ or as goal-coordinated and pure bimanual motions.³² Additionally, movements can be divided into translational and rotational movements. The analyzed dual-arm motions include contacts between hands and equipment. Thus, the arm motions are limited by the characteristics of the equipment and perform a primitive motion that includes translation/rotation around one axis. Based on the above-given classification and the axis of the rotation/translation, we have chosen different types of dual-arm motions

rotational motions—rotation of the steering wheel, rotation of valves, and rotation of the canoe paddles and

translational motions—cutting with a knife, inflating a mattress using a pump, grating food, and opening/closing a drawer.

Each recorded task consists of a five times repetition of a dual-arm motion primitive. Before recording, all actors practice each motion for approximately 5 min. Each actor carried out the required motions under the same conditions and using the same equipment. Nine male and six female actors (with age 32 ± 11 years; with the height 1.75 ± 0.1 m) participated in the experiments.

In order to detect the starting and ending point of one primitive motion, we paid attention to the samples where the minimum values of the joint velocity are detected and hands are currently stopped. One execution of a primitive motion is represented as a part of the joint trajectories between the samples of two minimum values, which are repeated during the motion. We normalized temporarily each primitive motion in order to have the common phase for performing the same motion for all actors.

Since the characteristics of the human motion (criterion optimized) are obtained by imitating the recorded human motion using our inverse optimal control algorithm and the criterion functions, the quality of imitation is defined through the value of the fitness function (see equation (13)). The fitness function is calculated using the genetic algorithm as described in the previous subsection. The fitness function, calculated for the best combination of the weight coefficients, represents the minimum deviation between the obtained and the recorded motion. According to the results of the genetic algorithm, the best combination of the basic criterion functions with the same value of the fitness function is obtained if more than 80% individuals in a generation converge to the same solution. The average values of the fitness function are calculated per sample for all actors. Table 1 presents the fitness function for all experiments.

Table 1.

The average values of the fitness function in a one-time sample of all actors for all tasks.

Task	Fitness (m)
Rotation of the valves	0.0466
Rotation of the canoe paddles	0.0413
Rotation of a steering wheel	0.0127
Inflating a mattress using a pump	0.0096
Cutting with a knife	0.0043
Grating of food	0.0029
Opening/closing a drawer	0.0206

The fitness function given in Table 1 represents the integral sum of the errors between the achieved and the recorded position of all observed joints (right elbow, right shoulder, left elbow, and left shoulder). Therefore, the average error in the following motion of each joint per sample in “opening/closing a drawer” task is about 0.0052 m, since the fitness function sums displacement of four joints, two elbows, and two shoulders. Hence, the conclusion is that each recorded human motion can be reproduced with great accuracy by the appropriate combination of criterion functions.

In the following subsections, the objective function (10) minimized for various observed motions is presented. A qualitative motion evaluation in joint space is considered to explain the obtained results. The final part of this section will provide a general conclusion on motion characteristics and their correlation with basic criterion functions.

“Opening/closing a drawer”

In this subsection, the characteristics of the “opening/closing a drawer” task are presented. The “opening/closing a drawer” task represents a symmetric horizontal-translation motion (see Figure 3(a)). The motion is constrained by the equipment and only horizontal translation of the hands is allowed. The orientation of the hands is fixed and prescribed by the equipment (the fingers of both hands are placed inside the drawer with the palms down). The height of hands is determined by the characteristics of the drawer.

Figure 3.

(a) “Opening/closing drawer” task. (b) Resulting weight coefficients defining the objective function—the criterion “minimization of joint velocity” prevails. (c) Joint motions—shoulder and elbow motions dominate.

Upon observing the trajectories of hand motion for the task, recorded by the motion capture system, it is obvious that the motions of all actors for the given task are close, mainly because it is highly conditioned by the characteristics of the equipment. The analysis of motion in joint space is performed using the results from the imitation algorithm, which give us trajectories of each DoF. The results show that during this motion, the shoulder (shoulder pitch and shoulder yaw) and elbow joints move more than other joints (see Figure 3(c)). Furthermore, the motion of the elbow yaw joints is larger than the motion of shoulder joints. The trajectories of the joints given in Figure 3(c) represent the mean values obtained for all actors during the same motion. We also noticed that the size of an actor affects the motion of arm joints during the task because the position of the equipment was fixed and was not determined by the size of the actor. In the case when the actor is taller, the actor’s arms are bent down. On the other hand, smaller actors did not bend down their arms and had to rely more on their elbows and trunk joints to perform the task.

The combination of the weight coefficients obtained by the genetic algorithm while solving the inverse optimal problem for the task “opening/closing a drawer” is given in Figure 3(b). The results of the genetic algorithm, obtained for all actors in the same motion, are depicted in the 3-D graph where horizontal axes show a number of actor and particular criterion, while the vertical axis shows the impact of a particular criterion per each actor. For descriptive and consistent illustration, impact of each criterion is highlighted in the same color

green—maximization of manipulability criterion,

magenta—ergonomy criterion (minimization of distance between the current and ergonomic configuration criterion),

red—minimization of joint velocity criterion, and

blue—minimization of kinetic energy criterion.

The results show that 9 of 15 actors use the velocity minimization criterion with the value k_velocity which takes values near 1. Since the shoulder and elbow joints have far greater motion than other joints, the velocity minimization criterion is dominant for such type of motion. In accordance with the results presented in Figure 3(b), the criterion of manipulability is dominant in the case of six actors when an actor has some restrictions on motion caused by its dimensions and/or distance from the drawer (indicated in green color in Figure 3(b)). In the case of four actors, the mean value of k_{manipulability} is 1, while in the case of two actors where the velocity criterion has an influence in generation motion, k_{manipulability} takes value 0.53. Some problems appear when an actor is not well positioned for performing the task. Since the “opening/closing a drawer” task is horizontal translation motion, the distance between actor and equipment has an influence on the way in which motion will be performed. If the actor is far from the drawer, he/she will keep the arms straight and try to perform the task. This arm configuration is near singularities and reduces the possibilities of arm manipulation. The actor tries to move hands away from the singularity and maximizes manipulability. The problem of manipulation also appears in the case when the actor is near the drawer. That is why the actor moves all joints more in order to increase manipulability and perform the task.

Eventually, one can conclude that for “opening/closing a drawer” task, human motion is planned so to minimize joint velocities and maximize manipulability, while their relative ratio depends on actor and equipment characteristics.

“Rotation of the valves”

In this subsection, the characteristics of the task “rotation of the valves” will be presented. The “rotation of the valves” is a goal-oriented rotational motion around the vertical axis (see Figure 4(a)). Both arms work independently while performing this task. Palms are placed perpendicularly to the room floor and grasp the handles vertically placed on the valves. The height of hands is determined by the characteristics of the valves.

Figure 4.

(a) “Rotation of the valves” task. (b) Resulting weight coefficients defining the objective function—the “criterion maximization of manipulability” prevails. (c) Joint motions—shoulder and elbow motions dominate.

According to the analysis of the motion in the joint space, it is obvious that some common characteristics could be observed for “rotation of the valves” and “opening/closing a drawer” tasks. The results obtained in the joint space (see Figure 4(c)) show that during these motions, the shoulder (shoulder pitch and shoulder yaw) and elbow joints move more than other joints. Thus, it is also expected that minimization of joint velocity criterion is dominant for the “rotation of the valves” motion.

Although one can intuitively expect that minimization of joint velocity dominates due to intensive joint movements, obtained results could be well justified and explained. The combination of the weight coefficients obtained by the genetic algorithm is presented in Figure 4(b) and shows that for this motion, the criterion maximization manipulability is dominant in the case of the nine actors. Mean value of k_{manipulability} in these cases is 0.9048 with variance 0.0218. The minimization of joint velocity criterion is dominant in the case of the four actors with a mean value of k_velocity equation 0.8517 and variance 0.0501. The minimization of distance between the current position and the ergonomic configuration is dominant in the case of two actors (in the case of actor no. 1, k_ergonomy = 1, while in the case of actor no. 14, k_ergonomy = 0.5267). The criterion maximization of manipulability is dominant when the position between actor and equipment or actor current pose is not suitable for performing the motion in a common way. The actor is forced to take a certain pose in order to accomplish the task easier. In the example of “rotation of the valves” motion, several cases appear

the actor is tall and far from the equipment—the actor bends the trunk (increase amplitude of the trunk pitch joint) and performs the task moving the shoulder and elbow joints and

the actor is small (or tall) and near the equipment—the actor keeps trunk vertically and moves shoulder and elbow joints more in order to perform the task.

Since the actor is far or near equipment, these body poses require additional movements to avoid singularities.

Human tendency to perform the task in the most comfortable manner is evident in the case of the two actors, for whom ergonomy criterion prevails. The comparison between the average configuration of the actor through the motion and the comfortable configuration shows that in the case of the one actor, these configurations are similar, while in another case, the influence of the maximization manipulability criterion is evident (and k_{manipulability} = 0.4204 in the case of actor no. 13).

According to the results, we can conclude that in the “rotation of the valves” motion, the position between actor and equipment and actor characteristics have a strong influence on criterion contribution. In a line with the analysis of the motion in the joint space, it is evident that the minimization of joint velocity criterion describes this motion best if the actor is well posed with the respect to the equipment and he/she is of medium height. In the cases when these conditions are not satisfied, the criterion minimization of distance between the current and ergonomic configuration and maximization of manipulability are dominant.

“Rotation of a steering wheel” and “inflating a mattress using a pump”

The “rotation of a steering wheel” task is an asymmetric dual-arm rotational motion (see Figure 5(a)). The motion of the hands is circular in accordance with the form of a steering wheel. The rotation starts from the initial position where the arms are symmetrically placed on the wheel. The hands are able to rotate the steering wheel (diameter 0.3 m) in both directions (in the experiments, the motion was ±90° starting from the initial position). During the motion, the relative position between both hands is unchangeable.

Figure 5.

(a) “Rotation of a steering wheel” task. (b) Resulting weight coefficients defining the objective function—the criterion “minimization of kinetic energy” prevails. (c) Joint motions—shoulder and wrist motions dominate.

The “inflating a mattress using a pump” task is a symmetric dual-arm translation motion (see Figure 6(a)). The hands grasp the equipment horizontally and their relative position does not change during the motion. In both motions, the actors are sitting while they carry out the tasks.

Figure 6.

(a) “Inflating a mattress using a pump” task. (b) Resulting weight coefficients defining the objective function—the criterion “minimization of kinetic energy” prevails. (c) Joint motions—elbow and wrist motions dominate.

During the “rotation of a steering wheel” task, the hands have a large motion in the Cartesian space and the task requires greater activation of shoulder and wrist joints compared to the motion of other joints (see Figure 5(c)). The wrist joints (wrist pitch and wrist roll) are active during the motion because the actors should change the hand orientation with respect to the referent coordinate system in order to perform the motion. As for this motion, the actors use their hands only and sit comfortably while using the equipment. The trunk motion is limited.

The results show that the criterion of kinetic energy minimization is dominant for this type of the motion (see Figure 5(b)). The influence of the inertia is evident in these results. The criterion of kinetic energy minimization is dominant in the case of the eight actors (the mean value of the weight coefficient k_energy is 0.8165 with variance 0.0116). The motion of the wrist joint does not have a big influence on the choice of the criterion functions because its inertia is significantly smaller compared with the inertia of the other joints (shoulder joints).

During the “rotation of a steering wheel” motion, the actor’s body position is near the human ergonomic position. This is evident in results, whereas in the case of five actors, the criterion of kinetic energy minimization shares its domination with ergonomy criterion (mean value of the weight coefficients k_ergonomy is 0.4738 and mean value of the k_energy is 0.4313). Two of the actors adapted the position of joints in order to decrease the motion of the shoulders. In this case, the velocity minimization criterion is dominant, which is also supported by the results of our inverse optimal control algorithm.

The angle of rotation of a steering wheel is not limited and it happened that some actors made a bigger angle of rotation compared to others. This fact confirms that the choice of criterion function is related to the activation of joints during the motion.

In the case of “inflating a mattress using a pump” task, elbows are the most active joints compared with other joints and the criterion of kinetic energy minimization is dominant in the case of nine actors as confirmed by the results presented in Figure 6(b) (mean value of the weight coefficients k_energy is 0.8896 with variance 0.0144). The pump produces greater resistance during the motion of the handle and additional effort was needed to perform the task. Besides the elbow, two actors used shoulder more intensively to perform the task and the criterion of the velocity minimization is dominant in their cases (weight coefficients k_velocity take the value 1). As well, the motions of some actors passed through the human ergonomic configuration and the criterion which minimizes distance between the current and the ergonomic configuration is dominant in two cases (mean value of the weight coefficients k_ergonomyy is 0.796 with variance 0.008). The criterion of manipulability appeared as dominant in the case of two actors (weight coefficients k_{manipulability} take the value 1) when the actors kept the arm straight and carried out the task moving the trunk.

The conclusion is that “rotation of a steering wheel” and “inflating a mattress using a pump” point out minimization of kinetic energy as a dominant criterion because the motion of a particular joint, especially the shoulder joint (for “rotation of a steering wheel”) and the elbow joint (for “inflating a mattress using a pump”), which have big effective inertia, moves more than other joints.

“Cutting with a knife” and “grating of food”

In this section, we analyzed “cutting with a knife” and “grating of food” together in order to compare the same type of the motion. During the motion, the left hand is used as a hand support, while the right hand performs the task.

The “cutting with a knife” task is one-arm support translation motion (see Figure 7(a)). The right hand does the translational motion in order to perform the task. The motion of the right hand is not strongly defined by the type of the equipment used. The right hand can rotate around the handle of the knife. The amplitudes of the right hand’s motion are limited by the size of the knife.

Figure 7.

(a) “Cutting with a knife” task. (b) Resulting weight coefficients defining the objective function—the criterion “minimization of joint velocity” prevails. (c) Joint motions—shoulder and elbow motions dominate.

The “grating of food” task is also a one-arm support translation motion (see Figure 8(a)). The orientation of the right hand is restricted and the palm should be in line with the plane surface of the grater. The trajectory of the right hand is related to the angle between the grater and the table surface, which is not predefined. Actors used grater in a way they considered the most comfortable. The right hand’s motion is limited by the size of the grater.

Figure 8.

(a) “Grating of food” task. (b) Resulting weight coefficients defining the objective function—the criterion “minimization of kinetic energy” prevails. (c) Joint motions—elbow motions dominate.

The “cutting with a knife” task is performed by the activation of the shoulder (shoulder pitch and shoulder yaw) and the elbow joints during the motion. The criterion of velocity minimization gives the same priorities to motions of these joints. This criterion is dominant in the cases of 10 actors. In accordance with the results obtained by the genetic algorithm, the mean value of the weight coefficient k_velocity for these cases is 0.9598 with the variance 0.0072.

The influence of the relative positions between actors and equipment and actor characteristics are showed up in this motion. The criterion of maximization manipulability is dominant in two cases (the weight coefficient k_{manipulability} takes the value 1 in both cases) when the actor is small and near the equipment and in the case when the actor is tall and far from the equipment. Therefore, the same motion planning pattern from the “rotation of the valves” tasks appeared in these motions. Since the actors were free to perform the task in the most comfortable way for them in the case of the two actors, the ergonomy criterion is dominant (in the case of actor no. 4, k_ergonomy = 1 and in the case of actor no. 10, k_ergonomy = 0.5632). For these actors, further analysis showed that the average position of the actor’s right-hand joints during the motion is near the human ergonomic configuration.

In the “grating of food” task, the motion of the elbow joint is dominant compared to other joints, which is shown in Figure 8(c). The criterion of kinetic energy minimization is dominant in this motion, which is supported by greater values of the weight coefficient k_energy in the case of 10 actors (the mean value of the k_energy values is 0.8875 and variance 0.0194). These results are expected since the motion of the elbow joint is greater than the movements of others joints (such as the case of the “inflating a mattress using a pump” task too). The other criteria are dominant in some particular cases. The criterion of maximization manipulability is dominant in the case of actor no. 12, while the criterion of minimization of distance between the current and the ergonomic configuration is dominant in four cases (the mean value of the k_ergonomy values is 0.9743 and variance 0.0015). Since the “grating of food” and “cutting with a knife” tasks are the one-arm support motions, the same conclusion for these exceptions stands.

“Rotation of the canoe paddles”

The “rotation of the canoe paddles” task represents a goal coordinated rotational motion around one horizontal axis (see Figure 9(a)). The relative position between the arms is constant and determined by the characteristics of the equipment. Palms of the hands are kept parallel to the room floor. According to the motion analysis for each joint, we can see that for this task the shoulder (shoulder pitch) and elbow joints have the big motion amplitude.

Figure 9.

(a) “Rotation of the canoe paddles” task. (b) Resulting weight coefficients defining the objective function—the criterion “minimization of joint velocity” prevails, although all other criteria are present depending on the actor’s characteristic. (c) Joint motions—shoulder and elbow motions dominate.

This motion requires motion of many joints to perform the rotational hand motion in the sagittal plane at the same distance independent of the actor in the frontal plane. Consequently, domination of different criteria depends a lot on the actor’s body characteristics. Intensive joint motions lead to domination of velocity minimization criterion in the case of five actors (the mean value of the k_velocity values is 0.8581 and variance 0.0489). On the other hand, the other criteria also contribute to this motion. The criterion of maximization manipulability appeared as a dominant in the case of three actors (the mean value of the k_{manipulability} values is 0.8823 and variance 0.0109), the criterion of minimization of kinetic energy is dominant in the case of three actors (the mean value of the k_energy values is 0.7132 and variance 0.0196), and the criterion of the minimization of distance between the current and the ergonomic configuration is dominant in the case of three actors (the value of the k_ergonomy is near 1), while the combination of all criteria is evident in the cases of one actor. It is very difficult to conclude which of the criterion function represented this motion best but, the relative position between actor and equipment as well as the human characteristics definitively has significant influence to the criteria which will be selected.

Discussion

The results reveal that the amplitude of each particular joint and a combination of joint activations during the motion influence the domination of the corresponding criterion function. In the case where the joints with big effective inertia are moved more compared to other joints, the human body needs more effort and energy to perform the motion. Thus, minimization of kinetic energy criterion dominates and the value of the weight coefficient k_energy is the largest compared to the values of other weight coefficients. In the tasks where the motions of all joints are with small amplitudes, the human does not waste a lot of energy during the task and consequently, minimization of velocity criterion is dominant, while the weight coefficient k_velocity has the biggest value.

The influence of inertia matrices is well presented in the task “rotation of a steering wheel.” The motion is obtained by moving the shoulder and wrist joints and the minimization of velocity criterion could be expected to be dominant because the amplitude of the motion of both joints for performing the task is similar. However, this is not the case because the effective inertia of the wrist joint is negligible compared to the inertia of the shoulder and the criterion of kinetic energy minimization is dominant. We can conclude that the choice of the criterion function and the motion strategy is highly related to the type of motion.

Moreover, the environmental characteristics, such as the size of the human body and the distance between the human and the environment, also affect the choice of criteria. Accordingly, it is expected that the same motion in the task space, performed by several people, can show different characteristics in the joint space. The criterion of manipulability appears as dominant for each task where the positions of the hands are close to a singular position or when joint motions are near the joint limits. The actor should adapt his/her motion in order to perform the task. The criterion of maximization of manipulability is the most expressed in the task “rotation of the valves.” The fact that a human performs a task in the comfortable (ergonomic) manner is proven in the cases of the several motions. The ergonomy criterion (minimization of the distance between the current and the ergonomic configuration) dominates in the majority of motions where average joint positions are near the human ergonomic configuration. The influence of this criterion is evident for the task “rotation of a steering wheel” since the equipment size and position are defined to be comfortable for the human. To sum up, a human will use a specific strategy (combination of the criterion functions) to perform the same task in the different environment provided that he/she is positioned well while performing the task. In some tasks, the choice of the criterion will be additionally defined by the characteristics of the actor.

The results obtained from the genetic algorithm show that the best imitation of human motion (minimal value of the fitness function) is not obtained in the case when only the dominant criterion is included in the IKs algorithm but the weighted combination of the criterion functions. The influence of each criterion separately and the combination of the criteria in the IKs algorithm without using the weight coefficients were earlier presented in article.²⁸ In some cases, the best imitation is obtained with the combination of all criterion functions. The similarity of the recorded actors’ motion and motions obtained for different values of weight coefficients for the “rotation of the canoe paddles” task is represented through the value of the fitness function. The average values of the fitness function in a one-time sample obtained by each criterion and by the optimal combination of the weigh coefficients (combination of all criteria) are represented in Figure 10. According to the results presented in Figure 10, we can see that the optimal combination of the weight coefficient gives the smallest values of the fitness function for number of actors (actors nos. 3, 4, 5, 6, 7, 8, 12, and 14) compared with the value of the fitness function obtained by each criterion. It implies that the optimal combination of the weight coefficient obtained by genetic algorithm generates the motion which is closest to recorded actor’s motion. On the other side, the genetic algorithm gave the results where the one criterion is absolutely dominant (the value of one weight coefficient is near 1) in the case of the actors nos. 1, 2, 9, 10, 11, 13, and 15. Therefore, in these cases, the combination of the weighted coefficients did not decrease the value of the fitness function and the best imitation of the recorded actor motion is obtained using one criterion. This is confirmed with the comparative analysis presented in Figure 10. However, changes in the value of some weight coefficients, even those are not dominant, in some cases, can greatly affect the quality of imitation and decrease the value of the fitness function.

Figure 10.

The average values of the fitness function in a one-time sample obtained by the each criterion and by the optimal combination of the weigh coefficients for the “Rotation of the canoe paddles” tasks.

Human-like dual-arm motion of the robot ROMEO

In the previous section, we defined the optimization algorithm for characterization of human motion. The algorithm is based on the inverse optimal control approach and criterion functions, which are able to describe human-like motions. The obtained results of the inverse optimal control algorithm are included in the IKs algorithm given by equation (12).

The following paragraphs show that our inverse kinematics approach with the optimal combination of the criterion functions is able to generate the human-like motions of the real humanoid robot ROMEO. The experimental validation is presented in this section.

The kinematic structure of robot hands is similar to a human and allows the robot to faithfully imitate all human motions. Furthermore, the distribution of segment masses of the robot ROMEO coincides with the human and it can be expected that the criterion functions which describe the motions of human arms will be the same for the motion of the robot. An additional condition that occurs is that the motion of the robot must be defined according to the characteristics of the robot (the length of segments and restrictions in the joints). In order to obtain human-like dual-arm manipulation task performed by the robot, the original motion of human hands should still be recognizable in the robot motion, although it has been modified according to the characteristics of the robot.

The knowledge of the weights of each criterion for the task will help us to define the generalized combination of criterion functions for each motion, which will eliminate the exceptions based on the position between the actor and the equipment and the characteristics of the actor. We additionally defined by the genetic algorithm which calculates the combination of weight coefficients and minimizes the sum of the fitness function of all actors while performing the same task. The results are presented in Table 2.

Table 2.

The generalized combination of criterion functions calculated for each experiment.

Task/criterion functions	Rotation of the valves	Rotation of the canoe paddles	Rotation of a steering wheel	Inflating a mattress using a pump	Cutting with a knife	Grating of food	Opening/closing a drawer
k _energy	0	0.2862	0.4448	0.7987	0	0.4384	0
k _velocity	0.3916	0.2223	0.0683	0.1942	1	0.0942	0.9989
k _ergonomy	0	0.2833	0.4869	0	0	0.4673	0.0007
k _{manipulability}	0.6084	0.2082	0	0.0070	0	0.0001	0.0003

The IKs algorithm proposed by (12) with a calculated combination of the weight coefficients will produce the human-like motion of the robot ROMEO. Figure 11 shows the snapshots of the motion in “opening/closing a drawer” task performed by the actor and the robot ROMEO. The motion of the robot shown in Figure 11(b) is obtained by our IKs algorithm with the generalized combination of the weight coefficients (presented in Table 2), while the motion of the robot shown in Figure 11(c) is obtained for the combination of the weight coefficients $k_{energy} = 1; k_{velocity} = k_{manipulability} = k_{ergonomy} = 0$ . The joint limits are included in the IKs algorithm using the approach explained in detail by Baerlocher and Boulic.³³ The motion of the robot hands is free of the collision with the equipment. The self-collision is avoided since the robot imitates the recorded human motion which is out of the self-collision. The images show that the motion of the robot obtained for the generalized combination of the weight coefficient tends to be more similar to the actor motion, compared to the motion of the robot obtained with the criterion minimization of the kinetic energy.

Figure 11.

“Opening/closing a drawer” task performed by the actor and the robot ROMEO. The motion of the robot obtained for the generalized combination of the weight coefficients (b) tends to be more similar to the actor motion (a), compared to the motion of the robot obtained with the criterion minimization of the kinetic energy (c).

The robot motion generated by the generalized combination of the weight coefficient produced motions in shoulder and elbow joints, as can be seen in Figure 11(b). The obtained motion of the robot is similar to the actor motion (see Figure 11(a)) and fully resembles the human motion. On the other side, the robot motion obtained by minimization of the kinetic energy is characterized by a large movement of the elbow joints, while the motion of shoulders is insignificant. The robot performs the task by moving the elbows to the side, away from the trunk. Moreover, we can see that upper arm in scenario (b) is aligned with the vertical axis as it is the case with the actor’s motion, while in scenario (c), it is not the case. Therefore, the obtained motion in (c) is not like the actor’s motion. These results are confirmed by the similarity measure. The similarity measure between the recorded actor’s motion (expressed as the motion of the scaled model of the robot ROMEO using imitation process ${\dot{q}}_{imitation}$ ) and the obtained motion of the robot ROMEO (using the IKs algorithm and weight coefficients ${\dot{q}}_{ROBOT}$ ) is calculated as a sum squared error over all joints velocities

S = \sum_{n = 1}^{N} {({\dot{q}}_{imitation} (t_{n}) - {\dot{q}}_{ROBOT} (t_{n}))}^{2}

The similarity measure of the robot’s motion obtained from the generalized combination of the weight coefficient is 0.0269, while the similarity measure for the robot motion obtained by minimization of kinetic energy criterion is 0.0762. The conclusion is that the IKs algorithm given by (12) with the combination of the weight coefficients, obtained from our inverse optimal control algorithm, can generate the same human-like motion with a humanoid robot with the kinematic structure close to or resembling that of the human body.

Conclusion and the future work

The present study presents the inverse optimal control algorithm as the optimization tool for the analysis of the characteristics of the basic dual-arm human motion using the combination of the basic criterion functions. The study is performed on the set of seven basic human motions performed by 15 actors. The obtained results provide general conclusions on human motion, as follows:

The characteristics of dual-arm motions performed by a healthy human are directly connected with the activation of the particular arm joints and a combination of joint activations since these humans try to do motions in the way they consider most comfortable (optimal).

The criterion of kinetic energy minimization is a dominant criterion for the tasks that require greater mobility of the shoulder, elbow, or trunk joints (the joints with bigger values of the inertia matrix) but not wrist joint.

Tasks that are not characterized by a large motion of the joints or evenly activation of the joints have a dominant criterion of minimization of velocity.

In tasks where the human performs motion near the singular configuration or near his joint limits, the criterion of manipulability minimization is dominant.

In each of the analyzed motions which pass near human ergonomic configuration, the criterion minimization of the distance between the current and the ergonomic configuration is dominant since humans will perform the motion on the most comfortable way if it is possible according to the characteristics of the task.

The optimal function (using weights of basic criterion function) exists. Changes in the value of some weight coefficients even those which are not dominant significantly affect the quality of the imitation and increase the value of the fitness function.

The strategy of performing the same motion by different actors is the same but may change due to the influence of the environment and human body characteristics.

Our IKs algorithm is able to generate the human-like motions with a redundant humanoid robot with the kinematic characteristics close to or resembling those of humans. The IKs algorithm with different combinations of weight coefficients will produce the different robot’s motions in joint space for the same motion of the robot’s hands in the task space. Our IKs algorithm with a generalized combination of weight coefficients (calculated for each task separately) will produce the robot’s motions which are closest to recorded human motions.

The results of the research can be applied to several areas. The characteristics of the basic motion of healthy people acquired in this work can be used for the analysis of human motion with the disability in motor skills. Furthermore, our IKs algorithm can be used for generation of a complex motion, which represents a set of the analyzed basic motions, changing the combination of the weight coefficients from one basic motion to another. Implementation of the explored characteristics of human motion on the humanoid robot will enable the most natural cooperation between humanoids and humans, help the elderly persons in their everyday life, and allow better integration of humanoid robots into the human environment.

Our future research should be directed toward enlarging the set of the analyzed basic human motions and inclusion of basic criterion functions which consider dynamics. The soft computing methods, such as fuzzy logic, will be implemented to calculate the characteristics of new human motion (the weights), which will be used in the IK model to generate human-like humanoid motion.

Footnotes

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the Ministry of Education, Science and Technological Development, Republic of Serbia, under contract TR-35003, and Equipex Robotex project by CNRS under grant agreement no ANR-10-EQPX-44-01.

References

Klette

Tee

. Understanding human motion: a historic review. Comput Imaging Vision 2008; 36: 1.

Calinon

Billard

. Stochastic gesture production and recognition model for a humanoid robot. In: Proceedings of the IEEE/RSJ international conference on intelligent robots and systems (IROS), Sendai, Japan, 28 September–2 October 2004, pp. 2769–2774. IEEE.

Calinon

Billard

. Incremental learning of gestures by imitation in a humanoid robot. In: Proceedings of the ACM/IEEE international conference on human-robot interaction, Arlington, VI, USA, 10–12 March 2007, pp. 255–262. New York, NY, USA: ACM.

Calinon

Guenter

Billard

. Goal-directed imitation in a humanoid robot. In: Proceedings of the IEEE international conference on robotics and automation (ICRA), Barcelona, Spain, 18–22 April 2005, pp. 299–304. IEEE.

Alexander

. Walking and running: legs and leg movements are subtly adapted to minimize the energy costs of locomotion. Am Sci 1984; 72(4): 348–354.

Mombaur

Laumond

JP,

Yoshida

. An optimal control model unifying holonomic and nonholonomic walking. In: Proceedings of the IEEE-RAS international conference humanoid robots (Humanoids), Daejeon, South Korea, 1–3 December 2008, pp. 646–653. IEEE.

Mombaur

Truong

Laumond

. From human to humanoid locomotion inverse optimal control approach. Auton Robot 2010; 28(3): 369–383.

Potkonjak

Popović

Lazarević

. Redundancy problem in writing: from human to anthropomorphic robot arm. IEEE Trans Syst Man CybernB (Cybern) 1998; 28(6): 790–805.

Chevallereau

Khalil

. A new method for the solution of the inverse kinematics of redundant robots. In: Proceedings of the IEEE international conference on robotics and automation (ICRA), Philadelphia, PA, USA, 28 April 1988, pp. 37–42. IEEE.

10.

Chiaverini

Oriolo

Walker

. Kinematically redundant manipulators. In: Siciliano

Khatib

(eds) Springer handbook of robotics. Berlin, Heidelberg: Springer, 2008; pp. 245–268.

11.

Pamanes

Zeghloul

. Optimal placement of robotic manipulators using multiple kinematic criteria. In: Proceedings of the IEEE international conference on robotics and automation (ICRA), Sacramento, CA, USA, 9–11 April 1991, pp. 933–938. IEEE.

12.

Zeghloul

Pamanes-Garcia

. Multi-criteria optimal placement of robots in constrained environments. Robotica 1993; 11(2): 105–110.

13.

Tomić

Vassallo

Chevallereau

. Arm motions of a humanoid inspired by human motion. In: Pisla

Bleuler

Rodić

. (eds) New Trends in Medical and Service Robots. Switzerland: Springer International Publishing, 2016; Vol. 38, pp. 227–238.

14.

Khatib

Demircan

De Sapio

. Robotics-based synthesis of human motion. J Physiol Paris 2009; 103(3):211–219.

15.

Zheng

Yamane

. Human motion tracking control with strict contact force constraints for floating-base humanoid robots. In: Proceedings of the IEEE-RAS international conference on humanoid robots (Humanoids), Atlanta, GA, USA, 15–17 Oct 2013, pp. 34–41. IEEE.

16.

Park

Hong

Lee

. Generating optimal trajectory of humanoid arm that minimizes torque variation using differential dynamic programming. In: Proceedings of the IEEE international conference on robotics and automation (ICRA), St. Paul, MN, USA, 14–18 May 2012, pp. 1316–1321. IEEE.

17.

Albrecht

Ramirez-Amaro

Ruiz-Ugalde

. Imitating human reaching motions using physically inspired optimization principles. In: Proceedings of the IEEE-RAS international conference on humanoid robots (Humanoids), Bled, Slovenia, 26–28 October 2011, pp. 602–607. IEEE.

18.

Billard

Calinon

Guenter

. Discriminative and adaptive imitation in unimanual and bi-manual tasks. Robot Auton Syst 2006; 54(5): 370–384.

19.

Yang

Marler

Kim

. Multi-objective optimization for upper body posture prediction. In: Proceedings of the AIAA/ISSMO multidisciplinary analysis and optimization conference, Albany, NY, 30 August–1 September 2004, Vol. 30, pp. 4506.

20.

Liu

Hertzmann

Popović

. Learning physics-based motion style with nonlinear inverse optimization. ACM Trans Graph (TOG) 2005; 24(3):1071–1081.

21.

Yang

Ding

Yang

An algorithm for simulating human arm movement considering the comfort level. Simul Model Pract Theory 2005; 13(5): 437–449.

22.

Chablat

Bennis

. A new muscle fatigue and recovery model and its ergonomics application in human simulation. Virtual Phys Prototyp 2010; 5(3):123–137.

23.

Yoshikawa

. Analysis and control of robot manipulators with redundancy. In: Proceedings of the Robotics Research: the first international symposium, Cambridge, MA, USA, 1984, pp. 735–747. MIT Press.

24.

Lee

. Dual redundant arm configuration optimization with task-oriented dual arm manipulability. IEEE Trans Robot Autom 1989; 5(1): 78–97.

25.

Vahrenkamp

Asfour

Metta

. Manipulability analysis. In: Proceedings of the IEEE-RAS international conference on humanoid robots (Humanoids), Osaka, Japan, 29 November–1 December 2012, pp. 568–573. IEEE.

26.

Khalil

Kleinfinger

. A new geometric notation for open and closed-loop robots. In: Proceedings of the IEEE international conference on robotics and automation (ICRA), San Francisco, CA, USA, 7–10 April 1986, pp. 1174–1179. IEEE.

27.

Advanced Realtime Tracking GmbH, Motion Capture User Manual, version 2.0, July 2013.

28.

Tomić

Chevallereau

Potkonjak

. Submitted on 10.11.2016. Human to humanoid motion conversion for dual arm manipulation tasks. Robotica.

29.

Zhang

. A time-varying coefficient based manipulability maximizing scheme for motion control of redundant robots subject to varying joint velocity limits. Optimal Control Appl Methods 2013; 34(2):202–215.

30.

Goldberg

Holland

. Genetic algorithms and machine learning. Mach Learn 1988; 3(2): 95–99.

31.

Zőllner

Asfour

Dillmann

. Programming by demonstration: dual-arm manipulation tasks for humanoid robots. In: Proceedings of the IEEE/RSJ international conference on intelligent robots and systems(IROS), Sendai, Japan, 28 September–2 October 2004, pp. 479–484. IEEE.

32.

Krűger

Schreck

Surdilovic

. Dual arm robot for flexible and cooperative assembly. CIRP Ann Manuf Technol 2011; 60(1):5–8.

33.

Baerlocher

Boulic

. An inverse kinematics architecture enforcing an arbitrary number of strict priority levels. Vis Comput 2004; 20(6):402–417.