Calibrating intuitive and natural human–robot interaction and performance for power-assisted heavy object manipulation using cognition-based intelligent admittance control schemes

Abstract

In the first step, a one degree of freedom power assist robotic system is developed for lifting lightweight objects. Dynamics for human–robot co-manipulation is derived that includes human cognition, for example, weight perception. A novel admittance control scheme is derived using the weight perception–based dynamics. Human subjects lift a small-sized, lightweight object with the power assist robotic system. Human–robot interaction and system characteristics are analyzed. A comprehensive scheme is developed to evaluate the human–robot interaction and performance, and a constrained optimization algorithm is developed to determine the optimum human–robot interaction and performance. The results show that the inclusion of weight perception in the control helps achieve optimum human–robot interaction and performance for a set of hard constraints. In the second step, the same optimization algorithm and control scheme are used for lifting a heavy object with a multi-degree of freedom power assist robotic system. The results show that the human–robot interaction and performance for lifting the heavy object are not as good as that for lifting the lightweight object. Then, weight perception–based intelligent controls in the forms of model predictive control and vision-based variable admittance control are applied for lifting the heavy object. The results show that the intelligent controls enhance human–robot interaction and performance, help achieve optimum human–robot interaction and performance for a set of soft constraints, and produce similar human–robot interaction and performance as obtained for lifting the lightweight object. The human–robot interaction and performance for lifting the heavy object with power assist are treated as intuitive and natural because these are calibrated with those for lifting the lightweight object. The results also show that the variable admittance control outperforms the model predictive control. We also propose a method to adjust the variable admittance control for three degrees of freedom translational manipulation of heavy objects based on human intent recognition. The results are useful for developing controls of human friendly, high performance power assist robotic systems for heavy object manipulation in industries.

Keywords

Industrial robot power assist robot object manipulation human performance augmentation flexible automation cognition human-centered robot control human–robot interaction intent recognition robot vision HRI calibration admittance control predictive control

Introduction

Human workers in various industries (e.g. manufacturing and assembly, civil construction, timber/forestry, mining, military and rescue operations, transport and logistics) need to manipulate heavy objects/materials. However, worker’s muscle-powered manipulation is tiresome, and it can reduce productivity and cause injuries, back pains, musculoskeletal disorders, and so on in workers. On the contrary, fully automated manipulation is usually inflexible and/or less adaptable. We, in this dilemma, posit that the human-in-the-loop flexible automation systems such as the power assist robotic system (PARS) can be used comfortably for its advantages in terms of flexibility, adaptability, accuracy, intuitiveness, ease of use and robustness due to direct man–machine interface.^1,2 For a PARS, adaptability of the human through sharing haptic information along with the robot’s mechanical strength through sharing the power results in a human–robot system that is superior to an individual robot or a human.¹

Kazerooni first introduced the fundamental principles of a PARS where information and power are shared to ease load manipulation.¹ State-of-the-art research works on PARSs show numerous initiatives toward developing PARSs for manipulating objects and materials.^{2

–11} Niinuma et al.² proposed an overhead crane–type PARS for object manipulation and also compared its performance to that of fully automated conventional manipulation. Doi et al.³ proposed a hand crane–type PARS actuated on pneumatic principles for manipulation of objects. Hara⁴ proposed a mechanism for switching power-assisting semiautonomous control to fully automatic control for manipulating objects in horizontal direction. Yagi et al.⁵ tested power assist control for human’s upper arm exoskeleton based on the pneumatic actuation principle for handling heavy loads in agriculture. Dimeas et al.⁶ developed the ideas of neuro-type admittance control for a PARS for lifting heavy objects. Hara and Sankai⁷ built “hybrid assistive limb” for assisting workers to carry heavy materials. In the literature,^8,9 variable admittance controls (VACs) of PARSs were developed for manipulating large objects. Numerous industrial assistive devices (IADs) for object handling were introduced by Colgate et al.¹⁰ Olivier et al.¹¹ presented “Cobomanip”—a special type IAD to handle loads for industrial purposes. In addition, we find a few assistive and power amplification robotic devices in practical industrial applications for heavy load handling, for example, the power loader light¹² and Cobot.¹³ However, the existing PARSs for load manipulation still suffer from limitations as explained below:

First, a human user perceives low weight while manipulating (say, lifting) an object with power assist.¹ Before touching the objects, the human estimates the load force (vertical lifting force) to lift the object, which depends on the object weight visually perceived by the human.¹⁴ We know that the perceived haptic weight (the weight perceived after lifting the object) is less than the visually perceived weight (the weight perceived through vision before touching the object) for power-assisted manipulation.¹ Consequently, the load force applied to the object estimated through visual weight perception is larger than the actually required load force.¹⁵ This phenomenon can cause unexpected acceleration, lack of safety, accident, and instability.^1,15 The human–robot interaction (HRI) and hence the manipulation performance can be unsatisfactory, and user’s trust in the PARS can reduce.^16,17 If the user performs careful visual investigations of the loads before manipulating the loads to realize the differences between haptically and visually perceived weights, fatigue and cognitive workload can be imposed upon the user.

Several techniques have been adopted to solve the above problems with PARSs, for example, compensation of gravity in system dynamics.^2,6,10 However, the gravity compensation method has drawbacks such as the zero gravity can remove haptic perceptions fully or partly, and thus can result in loose kinesthetic manipulation.¹⁶ As a consequence, the maneuverability and safety can be hampered. As an alternative approach, virtual mass can be used in the dynamics.^4,8,9 However, basis of estimating the value of the virtual mass is usually not justified, which can resist the desired HRI and manipulation performance.¹⁶ As another alternative, a tentative model of load force can be adopted feed-forwardly with a belief that the load force may be easily adjusted once the human user gains substantial experiences.^3,4 However, estimation of load force for power-assisted manipulation is a cognitive phenomenon that largely depends on human user’s visual perception of weight of the lifted object. Thus, the harmful effects of the excessive load force can only be mitigated through cognitive solutions such as reflection of weight illusion in dynamics and controls of PARSs.^15,16 However, this issue is ignored in the state-of-the-art PARSs,^{2

–11} except in a few preliminary initiatives.^15,16,18,19

Second, it is a challenging issue to determine effective control methods for manipulating objects with power assist.²⁰ Friction, large inertia, and other dynamic and nonlinear effects are expected when manipulating heavy objects with power assist.^8,10 Such effects can be compensated as well as positional accuracy can be achieved through admittance controls.⁸ The admittance parameters (e.g. damping, virtual mass, and stiffness) can affect HRI and manipulation performance. For example, low admittance parameters require less load force to accelerate an object that results in low fatigue, but the precision can be low because the robot is highly reactive.⁸ For large admittance parameters, large load force is required to manipulate an object and the human feels more heaviness, which can cause fatigue and the object movement can be slow due to low acceleration, but this can help achieve more precise and smooth manipulation.⁸ However, investigations on selecting appropriate control methods for power-assisted manipulation of heavy objects in industries did not achieve much priority.¹⁵

Third, not only robotic parameters such as displacement, speed, power, and torque but also HRI and task performance need to be assessed and optimized to achieve human-friendly manipulation.^21,18 Objective evaluation is emphasized, but there are HRI criteria that can neither be ignored nor measured objectively.^15,16,18 We posit that the subjective evaluation can be complementary to objective evaluation in such cases.¹⁸ In addition, the evaluation of HRI should include both physical HRI (pHRI)^22,23 and cognitive HRI (cHRI).²⁴ However, a holistic scheme to evaluate HRI and performance is not available, and optimization strategies to produce optimum HRI and performance are still not proposed for object manipulation with power assist except preliminary initiatives taken by Rahman and Ikeura.¹⁸ Furthermore, these preliminary initiatives were not validated for heavy object manipulation with power assist.

Fourth, a human can feel natural and intuitive when he/she manually manipulates a lightweight object with either a power or a precision grip.^22,25,26 This is why, psychology experiments involving object manipulation usually use lightweight, small-sized objects so that the human can grip the small object with a precision or a power grip, and thus the human can feel natural and intuitive.^22,25,26 The principle of power-assisted manipulation is to produce the perception of reduced heaviness in human hand even when the human manipulates a heavy object with power assist.¹ However, in usual experimental setups for power-assisted manipulation of large and heavy objects,⁸ the human cannot grasp the whole object using a power grip, it is difficult to attach a force sensor to the object directly, and thus the human usually applies load force to the object via a handle. Thus, the human may not feel intuitive and natural. Furthermore, human–machine interaction features for control algorithms for power-assisted manipulation of heavy objects cannot be benchmarked with the human features captured in object manipulation in psychology experiments that use lightweight, small-sized objects.^14,27 Thus, it is difficult to guarantee that the control performance for heavy object manipulation with power assist can fit with human psychology and meet expected intuitiveness and naturalness. In this situation, we posit that lifting small and lightweight objects with power assist can be used to determine the control parameters that generate natural and intuitive HRI and performance, and then the HRI and performance for lightweight objects can be used to calibrate the HRI and performance for handling heavy objects with power assist.^27,28 We call this intuitive and natural HRI and performance calibration approach. Our experience suggests that human-centered intelligent controls such as model predictive control (MPC)²⁹ and VAC⁹ can be two good tools to tune this calibration. However, such approach has not been investigated.

Based on the above motivation and background, we determine two objectives for this article to be addressed in two steps. In the first step in sections “Development of the 1-DOF PARS,” “Cognition-based dynamics model and control system for the PARS,” “The evaluation scheme and the recruitment of subjects,” “Experiment 1: Evaluation of the FAC for lifting lightweight object,” and “Evaluation results for the FAC for lightweight object,” we introduce a method to include weight perception in robot dynamics and then derive fixed admittance control (FAC) algorithm for the PARS based on weight perception. We build a one degree of freedom (DOF) PARS for lifting small-sized, lightweight objects, evaluate the control algorithm using a comprehensive evaluation and optimization scheme, and determine the optimum HRI and manipulation performance for the control algorithm using the optimization scheme. In the second step in sections “Experiment 2: Evaluation of the FAC for lifting heavy object,” “Comparing the evaluation results for the FAC between manipulation of lightweight and heavy objects,” “Cognition-based intelligent control strategies,” “Experiment 3: Evaluating the cognition-based intelligent control strategies,” and “Results of experiment 3,” we implement the FAC algorithm for lifting large and heavy objects with 3-DOF PARS (Cartesian manipulator), enhance the intelligence level of the FAC using weight perception–based MPC and vision-based VAC, and determine intuitive and natural HRI and performance for lifting the heavy object through calibrating with the HRI and performance for lifting the lightweight object. We consider vertical lifting here because (i) lifting along vertical direction is commonly observed in industrial practices for object manipulation and (ii) human workers need higher power assistance as workers feel more weight of object during vertical lifting. We also adjust the VAC for 3-DOF Cartesian manipulation of the heavy object based on human intent recognition. The results are novel that can help develop the control systems of human-friendly power assist devices to manipulate heavy objects in industries through augmenting human performance.

Development of the 1-DOF PARS

We constructed a 1-DOF PARS for vertical lifting and lowering of small-sized, lightweight objects, as shown in Figure 1. As shown in Figure 1(a), we fixed a ball screw and an AC servomotor coaxially on a rectangular metal plate and then attached the plate vertically to a wall. We attached a force sensor (foil-type strain gauge) to the ball nut through a holder-like device made of wood. We made a rectangular box–type object by bending an aluminum sheet of thickness 0.0005 m, dimensions 0.06 × 0.05 × 0.12 m, and self-weight 0.016 kg. We attached two rectangular aluminum blocks with a hole in the center of each inside the box to help tie it to the holder, as shown in Figure 1(b). Our intention was that a human user could grip the box with a power grip and lift it with power assist, as shown in Figure 1(c). Figure 2 shows the experimental setup including major components and the communication networks.

Figure 1.

(a) Configuration of the 1-DOF PARS, (b) back of the object (box), and (c) a human intuitively lifts an object with the PARS. DOF: degree of freedom; PARS: power assist robotic system.

Figure 2.

Experimental setup, major mechanical and electronics components, and communication networks for the PARS for lifting small-sized, lightweight objects by human user. PARS: power assist robotic system.

As Figure 2 shows, the AC servo system consisted primarily of a servo driver (servo pack) that drives the AC servomotor and its controller. The servo driver receives a command signal in the form of voltage signal from the controller in the computer through a digital to analog (D/A) converter, amplifies the signal, and transmits electric current to the servomotor to produce motion (e.g. acceleration). A position sensor consisted of an encoder and a counter attached to the servomotor reports the object’s actual displacement back to the servo driver so that it can compare the actual displacement with the commanded displacement and correct the commanded signal (pulse signal) if an error is found in the displacement. A force sensor measures the force applied to the object by the human in the form of voltage signal, amplifies it by an amplifier, and sends it to the control system via an analog to digital converter. The human force contributes only motion (acceleration) to the lifted object.

Cognition-based dynamics model and control system for the PARS

The dynamics model

Based on Figure 3, the dynamics for vertically lifting an object with the PARS by a human user along the x-axis is expressed in equation (1), where f_ax denotes the actuating force of the robot along the x-axis direction (N), f_hx denotes the load force applied by a human along the x-axis direction (N), F_x denotes the friction force in a ball screw along the x-axis direction (N), g is the gravitational acceleration (m/s²), K is the viscosity of the linear slider (N.s.m⁻²), m is the virtual mass of the object (kg), and x_d is the desired displacement of the object lifted with the robot along the x-axis direction (m) if x is the actual displacement of the object lifted with the robot along the x-axis direction (m).

Figure 3.

The dynamics model for lifting an object by a human user with the PARS. PARS: power assist robotic system.

To resemble free motion for lifting the object with the PARS, we ignore a few force terms, for example, forces due to viscosity, friction, disturbances, reaction, and actuation.³⁰ The free motion dynamics helps the human feel the natural environment that the human experiences when lifting an object manually. Thus, the dynamics model simplifies as in equation (2)

m {\ddot{x}}_{d} + K \dot{x_{d}} + m g + F_{x} = f_{h x} + f_{a x}

m {\ddot{x}}_{d} + m g = f_{h x}

During manual manipulation (e.g. lifting), a human feed-forwardly estimates the object weight based on visual cues (e.g. size, shape, and texture) and uses this information to feed-forwardly estimate the manipulative forces (grip and load forces).¹⁴ Like manual lifting, if the human takes the visually estimated large weight (and hence the large mass value or the m value) into account to estimate the inertial ( $m {\ddot{x}}_{d})$ and gravitational (mg) forces to lift an object with power assist, the applied load force in equation (2) will also be large that will jeopardize the motion and hamper the safety.^15,18 We know that there is a mismatch between visual and haptic perceptions of weight of an object lifted with power assist.^1,15,25 Thus, if the same mass value is used to estimate the inertial and gravitational forces in equation (2), the tactile perception in human hand due to the gravitational force and the motion of the lifted object due to the inertial force may not be appropriate.¹⁸ Hence, it is necessary that (i) appropriate mass values are estimated to program the inertial and gravitational forces for the PARS and (ii) a mismatch in the mass values is considered for feed-forward estimation of inertial and gravitational forces so that such mismatch can counterbalance the effects of error on load force estimation due to difference in visual and haptic perceptions of object weight and can make the tactile perception and motion appropriate.¹⁶ To reflect such concept in the system dynamics, we adopted hypothesis 1, as follows:

H1: Haptic perception of object weight due to change in inertia may differ from haptically perceived weight due to change in gravity for power-assisted lifting.

To realize this hypothesis, we used the mass parameter of the inertial force, $m {\ddot{x}}_{d}$ , different from that for the gravitational force, mg. Hence, the dynamics model in equation (2) was modified as in equation (3). In equation (3), m ₁ and m ₂ are the mass parameters used to estimate the inertial and gravitational forces, respectively, and $m_{1} \neq m_{2} .$ Human cognition (weight perception) is fused into the system dynamics here, which brands it as a novel approach. The proposed approach of modeling system dynamics seems to be simple with respect to the state-of-the-art approaches of dynamics modeling for robotics systems.⁶ However, it is novel since human user’s cognition is taken into account, and it is also very useful for designing human-friendly robot controls.¹⁶

m_{1} {\ddot{x}}_{d} + m_{2} g = f_{h x}

The control system

We designed a feedback position control scheme for the PARS for lifting objects using the weight perception–based dynamics model in equation (3). The control scheme is shown in Figure 4. While implementing the scheme, the servo system is to be kept on the velocity control mode. The control input is the load force f_hx and the control output is the displacement of the lifted object x (and its derivatives). The commanded velocity of the object lifted with the robot along the x-axis direction (denoted by ${\dot{x}}_{c}$ ) and the displacement feedback are expressed in equation (4), where G is the gain of the feedback position control. Here, ${\dot{x}}_{c}$ is input to the servomotor through a D/A converter. The servomotor produces f_ax based on ${\dot{x}}_{c}$

{\dot{x}}_{c} = {\dot{x}}_{d} + G (x_{d} - x)

Figure 4.

Cognition (weight perception)-based novel FAC with position feedback and velocity controller for lifting objects with power assist. FAC: fixed admittance control.

The control scheme in Figure 4 resembles the admittance control, where force is usually the input and displacement is the output. This admittance control also integrates positional feedback and velocity controller.^8,9 Here, g is fixed. Thus, the system characteristics, HRI and manipulation performance will depend on m ₁, m ₂, G, and f_hx. The position-based admittance control is proposed due to its advantages in achieving positional accuracy and compensating friction, dynamic effects, and nonlinear forces.^8,20 We call the proposed control the FAC if m ₁, m ₂, and G are kept unchanged in a trial. Fusion of human cognition in the FAC makes it different and novel from the state-of-the-art power assist controls.^{2

–11}

The evaluation scheme and the recruitment of subjects

This section presents a scheme to evaluate the weight perception–based novel FAC (Figure 4). The scheme includes the evaluation of HRI and manipulation performance. We divided the HRI into two categories: pHRI and cHRI. The human–robot co-manipulation performance was expressed in terms of efficiency and precision. This section also introduces the subjects recruited to conduct the evaluations through various experiments.

The pHRI evaluation criteria and scale

We expressed the pHRI in a few terms as given in Table 1. We used a subjective rating scale to evaluate the pHRI based on these terms, as shown in Figure 5.

Table 1.

The pHRI evaluation criteria (terms) with descriptions.

Criterion	Descriptions
Motion (m_o)	Motion is evaluated through the nature and friendliness of the lifted object’s velocity and acceleration felt by user. For example, it may mean whether the velocity and acceleration are low or high compared to user’s expectation
Maneuverability (m_a)	Maneuverability is evaluated through user’s haptic perceptions (e.g. perceived heaviness, kinesthetic and tactile perceptions, proprioception) while lifting an object with the system
Naturalness (n_a)	Naturalness is evaluated through user’s overall likeability, convenience, non-complexity, and normalcy in operation and collaboration
Stability (s_t)	Stability of the human–robot system is evaluated through presence/absence of oscillations, sudden inactivity of the system, and so on and their effects on manipulation, object, system configuration, and environment
Health and safety (h_s)	Safety and user’s occupational health are evaluated through potential fatigue, injuries, accidents, impacts, and jerks on user’s musculoskeletal system

pHRI: physical human–robot interaction.

Figure 5.

The rating scale with possible scores (in parentheses) for evaluating the pHRI. pHRI: physical human–robot interaction.

The cHRI evaluation methods and metrics

We expressed the cHRI in terms of user’s cognitive workload and trust in the system. Trust is the willingness of the user to believe in the assistance provided by the robotic system.¹⁷ We used the NASA task load index to evaluate the cognitive workload.³¹ The users evaluated their trust subjectively using a Likert-type scale, as shown in Figure 6.³²

Figure 6.

The Likert-type scale with possible scores (in parentheses) to evaluate human user’s trust in the PARS. PARS: power assist robotic system.

Evaluation of human–robot manipulation performance

Deviation from the target displacement of the lifted object (e.g. displacement overshoot) was considered as a criterion to measure the precision objectively according to equation (5). In equation (5), P_t is the target position and P_m is the measured position. Efficiency was expressed as the ratio between the targeted co-manipulation time (T_t) and the measured co-manipulation time (T_m), as given in equation (6)

Precision = (1 - \frac{| P_{t} - P_{m} |}{P_{t}}) \times 100 %

Efficiency = (\frac{T_{t}}{T_{m}}) \times 100 %

The proposed evaluation scheme integrates the evaluation of system characteristics (e.g. kinetics—load force; kinematics—displacement, velocity, acceleration), HRI and manipulation performance. The measurement methods are both subjective (HRI) and objective (system characteristics and performance), which thus make it a comprehensive scheme.¹⁸ The scheme is novel as such comprehensive schemes are not used to evaluate the state-of-the-art PARSs.^{2

–11} The objective criteria were used because (i) these criteria represent the general features of power-assisted manipulation and (ii) the criteria are related to the key performance indicators (KPIs) of power-assisted manipulation in industries. Alternative objective measures can also be used: (i) computer vision using Kinect to monitor safety and precision, (ii) remote motion capture, and laser and ultrasonic position sensing to capture human kinematics, and (iii) Electromyography to estimate force exerted on user body and user fatigue. In addition, more KPIs can be sought.

Recruitment of subjects

We recruited six groups of human subjects (university researchers, staffs, and students), each consisting of 10 subjects as detailed in Table 2. No physical and mental problems were reported by the subjects. The subjects gave prior consents and they participated in the experiments voluntarily. The study was approved by the concerned ethical committee.

Table 2.

Statistics of the recruited human subjects.

Group	Gender	Age (years)
I	8 males, 2 females	23.34 ± 1.87
II	9 males, 1 female	24.63 ± 1.29
III	10 males	25.89 ± 2.32
IV	10 males	23.76 ± 1.67
V	7 males, 3 females	26.54 ± 2.19
VI	9 males, 1 female	25.46 ± 2.07

Experiment 1: Evaluation of the FAC for lifting lightweight object

Experimental objective

The objective of this experiment was to evaluate the novel FAC (Figure 4) for lifting small-sized, lightweight objects in terms of HRI and manipulation performance for various admittance control parameters (i.e. m ₁ and m ₂ values).

Control requirements

The requirements of the FAC were to produce satisfactory (optimum) HRI and manipulation performance.

Experimental design

We nominated 36 pairs of m ₁ and m ₂ values as our first guess. The values are given in Table 3. We did not consider m ₁ = 0 and m ₂ = 0 because zero inertia (m ₁ = 0) usually produces oscillations and instability and the human loses haptic senses partly at zero gravity (m ₂ = 0), which are not good for achieving satisfactory HRI and performance.¹⁶ The independent variables were the m ₁ and m ₂ values, and the dependent variables were (i) HRI (pHRI and cHRI), (ii) system characteristics (kinematics—displacement, velocity, acceleration; kinetics—load force), and (iii) manipulation performance (precision and efficiency).

Table 3.

Values of m ₁ and m ₂ (6 × 6 pairs) to be used in the experiment.

m ₁ (kg)		0.25	0.50	0.75	1.0	1.25	1.5
m ₂ (kg)		0.25	0.50	0.75	1.0	1.25	1.5

Experimental procedures and data records

Group I subjects separately participated in this experiment. We provided detailed instructions about the evaluation methods, criteria, and scales to the subjects. We implemented the FAC (Figure 4) for the PARS (Figure 1) using MATLAB/Simulink [version 2010b]. In each trial, we randomly selected a pair of m ₁ and m ₂ values from Table 3 and put in the FAC system with confidentiality and maintained G = −5.5. We ran the PARS, a subject lifted the object with the system up to a targeted height of 0.1 m (P_t in equation (5)) once, and then released the object (see Figure 1(c) for the detailed procedures). We instructed the subject to perform the lift within 3 s. Hence, the targeted time for a trial (T_t in equation (6)) was 3 s. We recorded the system characteristics separately after the trial. We asked the subject to evaluate the pHRI and cHRI following the evaluation scheme (“The evaluation scheme and the recruitment of subjects” section). We recorded the total time used for the lift using a stopwatch. In the same procedures, we conducted the experiment for each of the 36 pairs of m ₁ and m ₂ values for each of the 10 subjects separately.

Evaluation results for the FAC for lightweight object

Evaluation of the pHRI

We developed an objective function as in equation (7) and an optimization algorithm (Algorithm 1) to determine the m ₁ and m ₂ pair(s) that produced the optimum pHRI for a set of constraints. We considered two types of constraints: the hard constraints that were required to be satisfied and the soft constraints that were desired to be satisfied.¹⁸ We decided the constraints based on our experience. In equation (7), i is the subject, that is, i = 1, 2,…10. $c_{1}, c_{2}, \dots c_{5}$ are the constants (positive valued) that indicate the relative weight (importance) of the criteria. We used $c_{1} = c_{2} = c_{3} = c_{4} = c_{5} = 1$ , that is, all the criteria were to carry equal weight, though the values of the constants can vary depending on the importance of the criteria for particular applications.

Algorithm 1:

Determination of m ₁ and m ₂ pair with the highest J value.

Input score values for m_a, m_o, s_t, n_a, h_s

Archive the values of the evaluation criteria for 10 subjects in a file for each m ₁ and m ₂ pair (there are a total of 36 different files created for 36 pairs of m ₁ and m ₂)

Input $c_{1}, c_{2}, c_{3}, c_{4}, c_{5}$

Input hard and soft constraints:

m_a > 0, m_o > 0, s_t > 0, n_a > 0, h_s > 0 (hard constraints)

m_a > 2, m_o > 2, s_t > 2, n_a > 2, h_s > 2 (soft constraints)

Calculate J for each m ₁ and m ₂ pair

Return J values for the m ₁ and m ₂ pairs that pass the constraints

Sort the m ₁ and m ₂ pair with the highest J value

End

We have different evaluation criteria such as m_a, m_o, s_t, n_a, and h_s as in equation (7). We have 36 pairs of m ₁ and m ₂ values (Table 3) and 10 subjects for the experiment condition of the FAC with the lightweight object. We summed up the scores for m_a for all the 10 subjects for each m ₁ and m ₂ pair separately, which gave us the value of $\sum_{i = 1}^{10} m_{a}$ in equation (7) for each m ₁ and m ₂ pair. Similarly, we calculated the terms of equation (7) such as $\sum_{i = 1}^{10} m_{o}, \sum_{i = 1}^{10} s_{t}, \sum_{i = 1}^{10} n_{a}, \sum_{i = 1}^{10} h_{s}$ for each m ₁ and m ₂ pair. Then, we summed up $\sum_{i = 1}^{10} m_{a},$ $\sum_{i = 1}^{10} m_{o}, \sum_{i = 1}^{10} s_{t}, \sum_{i = 1}^{10} n_{a}, \sum_{i = 1}^{10} h_{s}$ as in equation (7) for each m ₁ and m ₂ pair and got J value for each m ₁ and m ₂ pair. We then determined the m ₁ and m ₂ pair with the highest J value that passed the constraints, which was the m ₁ and m ₂ pair producing the optimum pHRI

J (m_{1}, m_{2}) = c_{1} \sum_{i = 1}^{10} m_{a} + c_{2} \sum_{i = 1}^{10} m_{o} + c_{3} \sum_{i = 1}^{10} s_{t} + c_{4} \sum_{i = 1}^{10} n_{a} + c_{5} \sum_{i = 1}^{10} h_{s}

After executing Algorithm 1, we did not find any J value returned for the elected m ₁ and m ₂ pairs for the soft constraints. However, we found the highest value of J for the hard constraints for m ₁ = 0.5 kg and m ₂ = 0.25 kg. Thus, we considered m ₁ = 0.5 kg and m ₂ = 0.25 kg as the admittance control parameters producing the optimum pHRI. The results thus justify the effectiveness of hypothesis 1. Note that the use of a larger range of values of m ₁ and m ₂ (e.g. $1.5 > m_{1} > 0$ and $1.5 > m_{2} > 0$ ) and bigger G value could be an alternative approach to search for more general optimization for the pHRI. However, it could be exhaustive and the bigger G value could also lower the system performance.

Remark 1

The proposed optimization algorithm is an exhaustive search method that may fall within the “local optimization” concept.³³ It can also fall under “single variable optimization,” which is a classical optimization method.³⁴ Here, the “optimum solution” actually means the “most feasible” or the “best” or the “maximum achievable/attainable” solution based on the evaluation in terms of some selected evaluation criteria. We, here, search the optimum/best control condition through the optimization algorithm. This is a search tool to find out the optimum solution based on the HRI evaluation results, but it does not adjust or impact the HRI. This may not be a global optimization, but this approach can be extended to a global optimization, and sometimes a local optimization can also be a global optimization.^33,34

Remark 2

Obviously, all 36 m ₁ and m ₂ pairs may not produce optimum HRI. It does not mean that the algorithm is unable to find a solution or it is because of the algorithm or of the nature of the problem itself. Instead, it may mean that the algorithm is perfect, but the m ₁ and m ₂ pairs are not able to produce optimum solution due to improper inertia and gravity compensation.

Remark 3

Appropriate optimization method to optimize HRI and performance in human–robot collaboration especially in PARSs for industrial applications is necessary. Imagine how we could sort out the optimum/best m ₁ and m ₂ pair out of 36 pairs without using such a method. From this point of view, the proposed optimization is definitely a useful contribution toward determining optimum HRI (and human–robot collaborative performance).

Remark 4

The proposed optimization algorithm is novel because such useful and necessary optimization methods are not used to optimize HRI (and performance) in state-of-the-art practices. However, the novelty is not in optimization theory but in optimization application. Formulation of the optimization problem, derivation of the objective function, identification of evaluation parameters and weights, measurements of the parameters and weights, determination of the constraints, and so on were customized for this particular application. Thus, such customization is novel that augments the application paradigm of the optimization theory.³⁴

Remark 5

Similar optimization approaches were used for other applications such as optimizing subtask allocation between human worker and collaborative robot in human–robot collaborative assembly in manufacturing.³⁵ In addition, a plethora of well-established classical optimization techniques are available in the state-of-the-art literatures.³⁴ However, many of these standard methods may not provide the most appropriate solution for the situation of HRI due to their complexity and lack of practicality, which motivated us to use the proposed search-based local optimization method.^33,34 It is not a concern why we did not use a complex and totally novel classical optimization method. Instead, the concern is how we used a simple but useful optimization method to find out an optimum/best solution for a very important practical industrial problem, which we usually do not observe elsewhere.

Evaluation of the cHRI

We followed the similar optimization procedures for the cHRI scores (workload, trust) as we employed for the pHRI. However, we did not find any J value returned for the m ₁ and m ₂ pairs for the soft constraints. For the hard constraints, we found the J with the lowest workload for m ₁ = 0.5 kg and m ₂ = 0.25 kg. In the same way, we found the J with the highest user trust in the PARS for m ₁ = 0.5 kg and m ₂ = 0.25 kg. Hence, we decided m ₁ = 0.5 kg and m ₂ = 0.25 kg as the FAC parameters that produced the optimum cHRI.

We conducted analyses of variance (ANOVAs) and found that variations in pHRI and cHRI between the subjects for the optimum FAC parameters (m ₁ = 0.5 kg and m ₂ = 0.25 kg) were statistically nonsignificant (p > 0.05 at each case). This shows the generality of the results. We posit that the inclusion of weight perception in the control helped achieve the optimum results for the HRI for the hard constraints, though further efforts might help achieve optimum HRI for the soft constraints.

Evaluation of the system characteristics

Typical system characteristics for lifting an object with the PARS by a subject for m ₁ = 0.5 kg and m ₂ = 0.25 kg are shown in Figure 7. We know that the load force required to lift an object with power assist should be a bit larger than the simulated weight used for the lifted object,¹⁴ which is m ₂ g = 2.45 N for our case. We compare the force profile in Figure 7 with this standard and see that the peak load force (PLF) is more than three times larger than the actually required load force. The peak acceleration is also large because the acceleration is proportional to the load force.¹⁴ We argue that the PLF and peak acceleration could be larger if we did not include weight perception in the FAC.^15,18 Again, the large PLF and acceleration might be responsible for hindering optimum HRI for the soft constraints.

Figure 7.

Typical kinematics and kinetics data for a subject for lifting a lightweight object with the PARS for the FAC for m ₁ = 0.5 kg and m ₂ = 0.25 kg. FAC: fixed admittance control; PARS: power assist robotic system.

Experiment 2: Evaluation of the FAC for lifting heavy object

Experimental objective

The objective of experiment 2 was to evaluate the FAC (Figure 4) for lifting heavy objects with power assist. We investigated how the FAC performed for lifting lightweight objects in experiment 1. In contrast to experiment 1, the purpose of experiment 2 was to investigate how the FAC performed for lifting heavy objects with power assist so that we could compare the results between lifting lightweight (experiment 1) and heavy (experiment 2) objects with power assist for the FAC.

Hypothesis

We adopted the following hypothesis:

H2: HRI and manipulation performance in power-assisted manipulation should be similar for lightweight and heavy object manipulation if the same FAC is used with the same optimum control parameters (m ₁ and m ₂ values).

Real robotic system for heavy object manipulation

We developed a 3-DOF (Cartesian/translational motions only) PARS, as shown in Figure 8, for heavy object manipulation. We considered vertical lifting (vertical DOF along the x-axis) only. The system consisted of linear manipulators actuated by servomotors. The object was tied to an object holder. A handle was tied to a force sensor, and it was also attached to the object holder. A human applied the load force ( $f_{h x})$ at the handle when lifting the object with the PARS. The object was kept on a soft surface (on a chair top) before it was lifted.

Figure 8.

A human vertically lifts a heavy object with the power assist device along the x-axis. The object can also be manipulated along the y- and z-axes horizontally.

Experimental procedures

The procedures for experiment 2 were same as those applied for experiment 1, but we used only m ₁ = 0.5 kg and m ₂ = 0.25 kg (instead of 12 pairs of m ₁ and m ₂ values) as the control parameters for the control system (FAC), as shown in Figure 4, and implemented the control for lifting a heavy object weighed 7.5 kg using the system, as shown in Figure 8, instead of lifting a lightweight object. Group II subjects participated in this experiment separately. The targeted manipulation height (P_t in equation (5)) was 0.5 m, and the targeted time for a trial (T_t in equation (6)) was 15 s. The control requirements, experiment design, evaluation scheme, and data recording were also the same as those applied for experiment 1.

Comparing the evaluation results for the FAC between manipulation of lightweight and heavy objects

In this section, we compare the evaluation results for the FAC for m ₁ = 0.5 kg and m ₂ = 0.25 kg for lifting the lightweight object as (Figure 1) in experiment 1 (we call it FAC with light object) to that for the FAC for m ₁ = 0.5 kg and m ₂ = 0.25 kg for lifting the heavy object using the system (Figure 8) in experiment 2 (we call it FAC with heavy object).

Comparison of system kinematics and kinetics

As Table 4 shows, the PLF and peak acceleration are larger for the heavy object than that for the lightweight object. It happened because humans applied large load force for the large and heavy object.¹⁴ However, the PLF and peak acceleration could be larger if weight perception was not included in the control algorithm, that is, the differential effects of gravity and inertia considered in the control algorithm contributed to reduce/counterbalance the PLF and the resulting acceleration.^15,18,25

Table 4.

Mean (n = 10) PLF, peak acceleration, and peak velocity with standard deviations in parentheses for FAC with lightweight and heavy object.

FAC with	PLF (N)	Peak acceleration (m/s²)	Peak velocity (m/s)
Lightweight object	8.18 (0.67)	2.12 (0.12)	0.40 (0.07)
Heavy object	10.98 (0.81)	2.73 (0.18)	0.47 (0.04)

PLF: peak load force; FAC: fixed admittance control.

Comparison of HRI

Figures 9 to 11 show that pHRI and cHRI (workload and human’s trust in the robot) for the control with lightweight object are not similar as that for the control with heavy object. ANOVAs showed that variations in pHRI and cHRI scores between the controls for lightweight and heavy objects were statistically significant (p < 0.05 at each case). In both cases, the same control parameter values (m ₁ = 0.5 kg and m ₂ = 0.25 kg) were used that were supposed to produce similar haptic feelings, which could generate similar pHRI, especially maneuverability, safety, and stability. Cognitive workload for physical demand was supposed to be similar because similar m ₂ value usually results in similar haptic feelings.¹⁵ However, the PLF and the resulting acceleration were larger for the large and heavy object (Table 4). This happened because the human estimated the load force based on visual weight perception depending on perceived visual size of the object.¹⁵ We posit that the differences in PLF and acceleration for the heavy object actually overruled the effects of similarity on m ₁ and m ₂ values and resulted in different HRI between the FAC with light and heavy objects.

Figure 9.

Mean (n = 10) pHRI evaluation scores with standard deviations for the FAC with lightweight and heavy object manipulation. pHRI: physical human–robot interaction; FAC: fixed admittance control.

Figure 10.

Mean (n = 10) workload rating scores (out of 100) with standard deviations for the six dimensions of the NASA TLX for the FAC with lightweight and heavy object manipulation. FAC: fixed admittance control.

Figure 11.

Mean (n = 10) trust values with standard deviations for the FAC with lightweight and heavy object manipulation. FAC: fixed admittance control.

Comparison of co-manipulation performance

Efficiency: The mean required times for a trial (lifting and releasing time for an object) were determined separately for the FAC with lightweight and heavy objects. Then, the manipulation efficiency was calculated based on equation (6). Table 5 compares the mean efficiencies for the FAC with lightweight and heavy objects. The results show that the efficiencies for the FAC with lightweight and heavy objects are significantly different (p < 0.05 at each case). We believe that the difference in speed between the FAC with lightweight and heavy objects due to the difference in PLF (Table 4) resulted in different efficiency levels. Note that the target time depends on manipulation speed, operator’s own speed, and fatigue. High speed may adversely affect the HRI. Hence, an optimum trade-off between efficiency and HRI is needed.

Precision: The displacement profile in Figure 7 for a trial for the FAC with lightweight object shows that the displacement profile had multiple peaks. It means that the subject initially lifted the object with high velocity probably due to high load force and acceleration, then sent the object slightly downward to reduce the velocity and adjust with the effects of high acceleration, and then completed the manipulation with a reduced velocity. This strategy might help achieve precision but might also restrict higher efficiency due to the need of adjusting the motion. For the trials when the displacement was not multi-peaked, the efficiency might be higher but the precision might go down. The mean precision for the FAC with lightweight object for all trials was determined and compared with that for the FAC with heavy object as in Table 5. The results in Table 5 show that the efficiency and precision between the FAC with lightweight and heavy objects are statistically significantly different (p < 0.05 at each case). The differences in PLF, acceleration, and speed (Table 4) between the FAC with lightweight and heavy objects might be responsible for different efficiency and precision.

Table 5.

Mean (n = 10) co-manipulation performance with standard deviations in parentheses for the FAC with lightweight and heavy objects.

	Co-manipulation performance (%)
FAC with	Efficiency	Precision
Light object	94.29 (2.23)	95.84 (2.61)
Heavy object	89.02 (1.62)	90.07 (2.39)

FAC: fixed admittance control.

The above results show that hypothesis 2 is not justified yet, that is, HRI and performance for lifting heavy object are not yet calibrated to that for lifting lightweight object. We believe that suitable intelligent control strategies may be useful to reduce the PLF and acceleration (Table 4) for lifting heavy object that may make hypothesis 2 true.

Cognition-based intelligent control strategies

The objective of this section is to associate a few novel intelligent control techniques with the cognition-based FAC proposed in Figure 4 in order to make hypothesis 2 true.

Model predictive control

We formulated an MPC taking theoretical and conceptual inspiration from Rawlings and Mayne³⁶ and Kouvaritakis and Cannon³⁷ and practical inspiration from Rahman and Ikeura²⁹ and Rahman et al.³⁵ for the PARS, as shown in Figure 12. In Figure 12, $a (r u n)$ is the predicted optimal output (acceleration) in a run (lifting trial), a_r is the reference acceleration, and $u (r u n)$ is the predicted optimal input to produce $a (r u n)$ in the run. The MPC, as shown in Figure 12, can be implemented in conjunction with the FAC in Figure 4. In such a case, the MPC optimizer provides a predicted input $u (r u n)$ in real time in term of a voltage signal based on ${\dot{x}}_{c 1} = {\dot{x}}_{c} + a_{e} W_{e}$ , where ${\dot{x}}_{c}$ is given in equation (4), $a_{e} = a (r u n) - a_{r}$ is the difference between the output acceleration and the reference acceleration, and W_e is a weighing factor. The $u (r u n)$ generates an actuating force f_ax along the x-axis that generates a predicted optimal output acceleration $a (r u n)$ so that the difference between the output acceleration and the reference acceleration is minimized, which is expressed as a cost function in equation (8). A proper empirical mapping between f_ax and $a (r u n)$ is necessary for this purpose, which can be approximated 1.0:0.25 as observed in Table 4. The f_hx is also proportionated based on a nominated proportion so that a constraint in equation (9) is maintained (a resultant reduced force limit $f_{h x r}$ is maintained) at the time of lifting. Note that ${\dot{x}}_{c}$ of equation (4) is actually replaced by ${\dot{x}}_{c 1}$ and then fed to the PARS through the D/A, as shown in Figure 4.

Figure 12.

The proposed MPC in conjunction with the FAC for the PARS. MPC: model predictive control; FAC: fixed admittance control; PARS: power assist robotic system.

This approach can be comparable to the constant force approach for power assist manipulation.³⁸ In equation (9), W_h and W_a are other weighing factors. We estimate $a_{r} \approx 1.0$ m/s², $W_{e} = 1$ , $W_{h} = 0.5$ , $W_{a} = 0.5$ , and $f_{h x r} \approx 4.0$ N (it is much larger than simulated weight, 2.45 N¹⁴) empirically being motivated by the kinematics and kinetics data in Figure 7 and Table 4. The weighing factor values are the initially set values, and the optimizer can determine new values randomly to maintain the constraint and provide optimal solution. The proposed MPC can also be extended to manipulation along the y- and z-axes

min_{{\dot{x}}_{c 1}} J_{MPC} = a_{e} W_{e}

(W_{h} f_{h x} + W_{a} f_{a x}) \leq f_{h x r}

Remark 6

An MPC provides an optimal future control policy on some control horizon, taking into account predictions of the controlled variable(s) on some prediction horizon.^36,37 We, here, mentioned “values,” which may suggest unitary horizons. We have one input and one output. A horizon is a range of predicted inputs or outputs. Note that a unitary horizon can be a part of a large prediction horizon (range). Whether the inputs and outputs are single values or horizons, it depends on the requirements and inputs and outputs of specific applications.³⁵ The used reference acceleration value is determined empirically to help predict an optimal output acceleration value, and this is a special case of a large prediction horizon in a miniature or in a simple form. Here, f_ax, $f_{h x r}$ , $a_{r},$ and $a (r u n)$ can actually mean peak actuating force, peak resultant force, peak reference acceleration, and peak output acceleration values, respectively, as illustrated in Figure 7. Alternatively, these could mean average values or ranges of values and so on. But, we used peak values depending on our requirements.

Variable admittance control

The VAC through variation in admittance parameters (m ₁ and/or m ₂) may augment the performance of the FAC in Figure 4. We learned from Rahman et al.¹⁵ that m ₁ does not affect weight perception, but it affects kinetics and kinematics (load force and acceleration); m ₂ affects weight perception, kinetics, and kinematics.¹⁵ Based on this information, we assume that variation in the virtual mass m ₁ may adjust the load force and acceleration for power-assisted manipulation.^9,15 The VAC may be formulated in such a way that the value of m ₁ exponentially declines from a large value to a small value (m ₁ = 0.5 kg) when the human manipulates an object with power assist and the displacement along the desired axis exceeds a threshold (e.g. xth). Reduction in m ₁ will proportionally reduce the PLF and resulting acceleration and improve the HRI and performance, but will not adversely affect human’s haptic feelings because the human will not feel the change of m _1. ¹⁵ This concept can be modeled as an exponential decay function of m ₁ as in equation (10), where t is the decay time, m ₀ is the initial value of m ₁, and α is the decay constant. The value of m ₂ in equation (11) does not vary

m_{1} (t) = m_{0} \times e^{- α t} + 0.5

m_{2} = 0.25 kg

This VAC is novel in comparison with the state-of-the-art VACs⁹ because in our case only the inertial mass m ₁ varies keeping the gravitational mass m ₂ fixed, which modulates kinematics and kinetics keeping haptic perceptions unaltered.

Vision-based intelligent VAC for manipulation of heavy objects of different sizes

The load force f_hx influences the acceleration that affects the HRI and performance. Object’s visual size is the cue that determines the extent of the f_hx.¹⁴ On the other hand, the extent of reduction of PLF through the VAC, as in equation (10), depends on m ₀ and α. It means that the larger the object size is, it is better to use larger values of m ₀ and α. This is why, we here developed a vision-based intelligent approach to classify/identify the size of the target objects and automatically adjust the values of m ₀ and α of the VAC for objects of different sizes.

We considered three target object sizes: large, medium, and small. To facilitate the vision-based detection, we attached three rectangular pieces of paper of different sizes and colors (large-sized red paper, medium-sized blue paper, and small-sized green paper) on the large, medium, and small objects, respectively. A camera was set in such a way that the colored paper piece remained within its vision range during object manipulation with power assist (Figure 8). The camera took images of the object when the human attempted to manipulate it, and the images were processed automatically through an OpenCV library. The paper image was blurred utilizing a normalized box filter of the OpenCV and transformed from blue–green–red model to hue–saturation–value (HSV) model. This model helped identify the color of the paper. The OpenCV was applied to obtain all the parts of the colored paper image between certain threshold parameters of the HSV model. Contours were formed on the processed image based on border following algorithms,³⁹ and moments of area of the colored image were computed and compared with the reference. The identified color and size (moment area) of the paper attached on the object helped identify the corresponding size of the object. Both color and size cues were used under the sensor fusion concept for higher detection accuracy.⁴⁰ The concept is illustrated in Figure 13, where $A_{c m a}$ and $A_{r m a}$ are the computed and reference moments of area, respectively.

Figure 13.

Autonomous selection of VAC parameters for different sizes of objects using vision-based detection system. VAC: variable admittance control.

If the computed paper image area was similar as the reference within a range of tolerance and the paper color was blue, the object was the medium-sized object. If the computed area was larger than the reference and the color was red, the object was the large-sized object. Otherwise, the object was the small-sized object. Once the size of the object is determined, a predefined relevant set of values of m ₀ and α appropriate for the identified object size was then automatically selected and the VAC was executed in real time. This vision-based VAC model could also be extended to object manipulation along the y- and z-axes.

Multi-DOF manipulation based on human intent recognition

Trajectory for 6-DOF dexterous manipulation of objects with a power assist device is expressed by X in equation (12), where y, z, and x are the translational, and θ_y, θ_z, and θ_x are the rotational displacements along the y-, z-, and x-axes, respectively. In this article, we wanted to confine our focus to the translational manipulation, that is, we considered $X = {[y z x]}^{T}$ only. Hence, in addition to the vertical lifting of the object along the x-axis, we considered the left–right (along the y-axis) and the forward–backward (along the z-axis) object manipulation. The equations of motion for horizontal manipulation along the y- and z-axes are derived as equations (13) and (14), respectively, where f_hy and f_hz are the load forces applied along the y- and z-axes, respectively. It was proved by Rahman et al.⁴¹ that m ₁ = 0.5 kg and m ₂ = 0.03 kg provide optimum HRI and performance for horizontal manipulation

X = [\begin{matrix} y \\ z \\ x \\ θ_{y} \\ θ_{z} \\ θ_{x} \end{matrix}]

m_{1} {\ddot{y}}_{d} = f_{h y} + m_{2} g

m_{1} {\ddot{z}}_{d} = f_{h z} + m_{2} g

The equation of motion (dynamics model) for each axis direction helps determine the VAC for each axis. As we propose here, the dynamics models and the concerned VAC systems are adjusted automatically for different axes to provide optimum HRI and performance to all three axes. A change in the dynamics model when the human started to manipulate an object with the PARS was triggered in real time based on human’s intent recognition, which is illustrated in Figure 14, where $f_{h x t h}$ , $f_{h y t h}$ , and $f_{h z t h}$ are threshold load forces along the x-, y-, and z-axes that are empirically estimated as 0.5, 0.1, and 0.1 N, respectively, being inspired by the kinetics data in Figure 7. The load force direction at the beginning of manipulation is a cue that reflects human’s intent to manipulate along a particular axis.⁴² For example, when a human just started manipulation, if $f_{h x} > f_{h x t h}$ , it did mean that the human intended to manipulate the object along the x-axis, and hence the VAC based on the dynamics model appropriate for manipulation along the x-axis was executed, and so forth. In addition, automatic adjustment of m ₀ and α values for different object sizes, as shown in Figure 13, was executed to make the control robust and adaptive.

Figure 14.

Autonomous selection of system dynamics (and hence the associated VAC) for different manipulation axes based on human intent recognition. VAC: variable admittance control.

Experiment 3: Evaluating the cognition-based intelligent control strategies

The objective of experiment 3 was to evaluate the effectiveness of the proposed cognition-based control strategies. The experiment had two phases. In the first phase, we used four experiment protocols as follows:

MPC with light object: We implemented the MPC, as shown in Figure 12, in conjunction with the FAC in Figure 4 for vertical lifting (x-axis direction) of a lightweight object using the PARS, as shown in Figure 1. Group III subjects participated in this experiment separately. The experiment procedures were the same as those used for experiment 1.

MPC with heavy object: We implemented the MPC in Figure 12 in conjunction with the FAC in Figure 4 for vertical lifting (x-axis direction) of a heavy object (7.5 kg) using the PARS, as shown in Figure 8. Group IV subjects participated in this experiment separately. The experiment procedures were the same as those used for experiment 1.

VAC with light object: We implemented the VAC derived from equation (10) in conjunction with the FAC in Figure 4 for vertical lifting (x-axis direction) of a lightweight object using the PARS, as shown in Figure 1. We used $x_{th} = 0.01 m$ , $m_{0} = 1.5 kg$ , and $α = 0.5 s^{- 1}$ as the VAC parameters. Group V subjects participated in this experiment separately. The experiment procedures were the same as those used for experiment 1.

VAC with heavy object: We implemented the VAC derived from equation (10) in conjunction with the FAC in Figure 4 for vertical lifting (x-axis direction) of heavy objects using the PARS, as shown in Figure 8. There were three heavy objects of different sizes (small 5 kg, medium 7.5 kg, and large 10 kg). Group VI subjects participated in this experiment separately. For each subject, the experimenter randomly selected a heavy object out of the three for lifting with the assist system. We nominated the VAC parameters as follows: (i) $m_{0} = 9 kg$ and $α = 3 s^{-1}$ for the large object, (ii) $m_{0} = 6 kg$ and $α = 2 s^{-1}$ for the medium object, and (iii) $m_{0} = 3 kg$ and $α = 1 s^{-1}$ for the small object empirically. We used $x_{th} = 0.05 m$ for all three cases. The autonomous selection of VAC parameters for different object sizes, as shown in Figure 13, was in effect. The experiment procedures were the same as those used for experiment 1.

In the second phase, the VAC in equation (10) was implemented in conjunction with the FAC in Figure 4 for manipulation of a heavy object (7.5 kg) using the PARS, as shown in Figure 8. Group I subjects participated in this experiment separately. The subject was allowed to manipulate the object along any of the x-, y-, or z-axes directions based on the subject’s own intent. The autonomous selection of system dynamics (and the relevant VAC) for different manipulation axes based on human intent recognition (Figure 14) was in effect. After the manipulation, we recorded the physically observed axis direction of the manipulation, which indicated the human’s intent. Then, we identified and recorded the actually executed control for the manipulation from the MATLAB/Simulink command window.

Results of experiment 3

At first, we analyzed the results of the first phase of experiment 3. Table 6 shows the PLF, peak accelerations, and peak velocity for different control strategies. The results show that both the MPC and the VAC reduced the PLF and peak accelerations (in comparison with that for the FAC in Table 4) for both lightweight and heavy objects. We see that the MPC was more effective in such reduction. However, the MPC reduced the velocity according to equation (4), which might reduce the efficiency. Again, the reduction in PLF and peak accelerations was slightly less for the heavy object though the difference was not statistically significant (p > 0.05 at each case). It was due to the reason that the human applied large load force if the object was large (heavy).¹⁴ We believe that the PLF and peak acceleration could be larger for heavy objects if the intelligent controls especially the VAC with autonomous adjustment of VAC parameters for different object sizes were not in effect. Even though the lifted object was heavy, the proposed intelligent control strategies calibrated the PLF and the resulting peak acceleration for lifting the heavy object to that for lifting the lightweight object, which is considered here as intuitive and natural.^22,26

Table 6.

Mean (n = 10) PLF, peak acceleration, and peak velocity with standard deviations in parentheses for controls with lightweight and heavy objects.

Control strategies	PLF (N)	Peak acceleration (m/s²)	Peak velocity (m/s)
MPC (light object)	4.11 (0.10)	1.23 (0.05)	0.22 (0.02)
MPC (heavy object)	4.16 (0.08)	1.26 (0.03)	0.24 (0.03)
VAC (light object)	4.48 (0.12)	1.35 (0.04)	0.40 (0.02)
VAC (heavy object)	4.52 (0.10)	1.38 (0.07)	0.41 (0.05)

PLF: peak load force; MPC: model predictive control; VAC: variable admittance control.

Figures 15 to 17 and Table 7 compare the pHRI, cHRI, (workload and trust) and manipulation performance among the control strategies. We see in Figure 15 that the pHRI for the heavy object was slightly lower than that for the lightweight object though the difference was not statistically significant (p > 0.05 at each case). The slightly larger PLF due to visual size effects and the resulting larger acceleration for the heavy object might be responsible for the slightly lower pHRI for the heavy object. However, maneuverability between lightweight and heavy objects was similar because maneuverability depends on gravitational mass m ₂,¹⁶ which was the same for both heavy and lightweight objects. The pHRI with lightweight object was considered natural and intuitive.^22,26 Hence, similar pHRI between lightweight and heavy objects generated by the controls indicates that the pHRI for the heavy object was calibrated with that for the lightweight object, and thus the pHRI for the heavy object was natural and intuitive.^22,26,27

Figure 15.

Mean (n = 10) pHRI scores with standard deviations for different control strategies. pHRI: physical human–robot interaction.

Figure 16.

Mean (n = 10) cognitive workload scores with standard deviations for different control strategies.

Figure 17.

Mean (n = 10) human trust in PARS with standard deviations for different control strategies. PARS: power assist robotic system.

Table 7.

Mean (n = 10) co-manipulation performance for controls with lightweight and heavy object.

	Value (%)
Control strategies	Efficiency	Precision
MPC (light object)	97.22 (2.31)	98.77 (2.52)
MPC (heavy object)	95.17 (1.47)	96.62 (2.11)
VAC (light object)	99.34 (2.36)	99.83 (2.39)
VAC (heavy object)	99.26 (1.78)	99.71 (2.06)

MPC: model predictive control; VAC: variable admittance control.

The main factor that determines pHRI is the load force and the resulting acceleration.¹⁶ Table 6 shows that the PLF and acceleration for MPC are lower than that for the VAC. Hence, the pHRI for the MPC should be better than that for the VAC. However, the results in Figure 15 are opposite. In fact, the MPC reduced the excess PLF (and acceleration), but at the same time it also affected the haptic perceptions adversely according to equation (3), which affected the maneuverability. Lower maneuverability lowered naturalness because both maneuverability and naturalness depend on haptic feelings.¹⁶ These problems did not happen for the VAC because the change in m ₁ as in equation (10) did not change the haptic feelings.¹⁵ Again, the change in haptic feelings for the MPC adversely affected human’s haptic engagement in manipulation, which resulted in loose haptic engagement between human hand and object, and thus human’s control over the object reduced, which reduced the safety and stability.¹⁶ The acceleration was lower for the MPC, but the velocity also reduced. For the VAC, the acceleration reduced, but the velocity did not reduce. As a result, the subject rated overall motion for the VAC over that for the MPC.

We see that the trends in pHRI for different control strategies (Figure 15) matched the trends in cHRI (Figures 16 and 17) and manipulation performance (Table 7). The results show that the VAC was more effective to produce higher HRI and performance for power-assisted manipulation than the MPC.^9,23,29 It happened because the VAC was empirical but more natural than the MPC. The MPC attempted to reduce the excess in the PLF through computing a predicted input to produce a predicted output based on optimization, which might not reflect human’s intent, and also the mapping between the input and the output might not be perfect due to nonlinearities in the PARS. On the other hand, the VAC took human intent (input load force that human naturally applies) into account and reduced the adverse effects of the excess of the load force in real time through the empirical relationship between m ₁ and PLF (and acceleration) without altering human’s haptic perceptions.¹⁵ It means that the model-based load force programming of the MPC did not exactly match with the natural (cognitive) force programming and perception (sensing) of the human in the VAC,⁴³ which resulted in slightly different results in terms of HRI and manipulation performance.

Again, similar results for the VAC between lightweight and heavy objects indicate that the VAC provided intuitive and natural HRI and manipulation performance for heavy object manipulation, which justifies hypothesis 2. Figure 15 shows that $m_{a} > 2$ , $m_{o} > 2$ , $s_{t} > 2$ , $n_{a} > 2$ , $h_{s} > 2$ for the VAC, which satisfy the soft constraints of the optimization in Algorithm 1. Similarly, the soft constraints of the optimization algorithm are also satisfied for the cHRI for the VAC. Hence, the natural and intuitive HRI and the resulting performance for the VAC were also the optimum for the soft constraints. The results show that calibration of HRI and manipulation performance for heavy object manipulation also improved the HRI and performance that helped achieve optimum HRI and performance satisfying the soft constraints.

Apparently, it may not be logical that the same control algorithm results in similar HRI and performance for power-assisted manipulation of lightweight and heavy objects. But it is logical because (1) control parameters were automatically adjusted for heavy objects and (2) actual weight of object was carried by the robot hardware, and the contribution of the control algorithm was to generate (i) haptic feelings in the human and (ii) motion. Note that, in this case, only haptic feelings and motion and the resulted HRI and performance were calibrated between lightweight and heavy objects; the physical structures for lightweight and heavy object manipulation were not calibrated.

Based on the results of the second phase of experiment 3, Table 8 shows the manipulation axis intended by the human subject and the dynamics model for the actually executed control system for each subject. The results show that the executed controls were appropriate for the attempted manipulation directions, which indicate the effectiveness of the autonomous selection of control systems for different manipulation axes based on human intent. The control system in each direction was intelligent admittance control based on weight perception. Hence, we believe that manipulation in all 3-DOFs resulted in optimum HRI and performance because the effectiveness of the intelligent controls for producing optimum HRI and performance for manipulation along the x-axis direction was proved as reflected in Figures 15 to 17 and Table 7.

Table 8.

Intended/attempted manipulation axis and the system dynamics for the actually executed control for different subjects.

Subject	Manipulation axis attempted by subject (physically observed by experimenter)	Dynamics model for the actually executed control system (identified in MATLAB/Simulink window)
1	Left-right (along the y-axis)	$m_{1} {\ddot{y}}_{d} = f_{h y} + m_{2} g$ , m ₁ = m ₀ e ^-αt + 0.5, m ₂ = 0.03 kg
2	Forward-backward (along the z-axis)	$m_{1} {\ddot{z}}_{d} = f_{h z} + m_{2} g$ , m ₁ = m ₀ e ^-αt + 0.5, m ₂ = 0.03 kg
3	Up-down (along the x-axis)	$m_{1} {\ddot{x}}_{d} = f_{h x} + m_{2} g$ , m ₁ = m ₀ e ^-αt + 0.5, m ₂ = 0.25 kg
4	Forward-backward (along the z-axis)	$m_{1} {\ddot{z}}_{d} = f_{h z} + m_{2} g$ , m ₁= m ₀ e ^-αt + 0.5, m ₂ = 0.03 kg
5	Forward-backward (along the z-axis)	$m_{1} {\ddot{z}}_{d} = f_{h z} + m_{2} g$ , m ₁= m ₀ e ^-αt + 0.5, m ₂ = 0.03 kg
6	Up-down (along the x-axis)	$m_{1} {\ddot{x}}_{d} = f_{h x} + m_{2} g$ , m ₁= m ₀ e ^-αt + 0.5, m ₂ = 0.25 kg
7	Left-right (along the y-axis)	$m_{1} {\ddot{y}}_{d} = f_{h y} + m_{2} g$ , m ₁= m ₀ e ^-αt + 0.5, m ₂ = 0.03 kg
8	Up-down (along the x-axis)	$m_{1} {\ddot{x}}_{d} = f_{h x} + m_{2} g$ , m ₁ = m ₀ e ^-αt + 0.5, m ₂ = 0.25 kg
9	Up-down (along the x-axis)	$m_{1} {\ddot{x}}_{d} = f_{h x} + m_{2} g$ , m ₁ = m ₀ e ^-αt + 0.5, m ₂ = 0.25 kg
10	Left-right (along the y-axis)	$m_{1} {\ddot{y}}_{d} = f_{h y} + m_{2} g$ , m ₁= m ₀ e ^-αt + 0.5, m ₂ = 0.03 kg

Conclusions and future works

A novel method to include weight perception in dynamics and control of a PARS for lifting objects was introduced, a comprehensive scheme for the evaluation and optimization of HRI and performance of the PARS was proposed, and the effectiveness of the weight perception–based control to achieve optimum HRI and performance for lifting lightweight objects was proved. Then, the effectiveness of the control and the optimization algorithm were validated for lifting heavy object with power assist. The HRI and performance for lifting heavy objects with power assist were calibrated with that for lifting lightweight power-assisted objects using a few intelligent control schemes such as MPC and VAC. Robot vision was used to adjust the intelligent VAC for objects of different sizes. In addition, load force-threshold-based human’s intent recognition was used to recognize the intended manipulation direction (axis), which adjusted the parameters of the proposed intelligent VAC. The findings proved two main concepts: (i) inclusion of weight perception in power assist control helps achieve optimum HRI and performance and (ii) the intuitive and natural HRI and performance can be calibrated from lightweight to heavy object manipulation using the intelligent controls, and the intelligent controls also improve HRI and performance. The optimization approach, PARS design, controls, calibration technique, evaluation scheme, and experimental results are fundamental and novel that can be particularly useful for the development of controls of user-friendly assist devices for heavy object manipulation in industries, for example, manufacturing, mining, construction, timber, and transport and logistics.

In the near future, we will search for advanced classical optimization methods and intelligent control strategies based on human features and expand the research for rotational DOFs for dexterous manipulation. We will also consider the PARS mounted on a mobile base so that it can be conveniently used to manipulate heavy objects in various locations in industry floor. We will verify and validate the findings in industrial scenario using robots actually performing industrial tasks with industry workers.

Footnotes

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

References

Kazerooni

. Extender: a case study for human-robot interaction via transfer of power and information signals. In: Proceedings of IEEE international workshop on robot and human communication, 1993, pp. 10–20.

Niinuma

Miyoshi

Terashima

. Evaluation of effectiveness of a power-assisted wire suspension system compared to conventional machine. In: Proceedings of IEEE international conference on mechatronics and automation, 2009, pp. 369–374.

Doi

Yamada

Ikemoto

. Simulation of a pneumatic hand crane power-assist system. J Robot Mechatron 2008; 20(6): 896–902.

Hara

. A smooth switching from power-assist control to automatic transfer control and its application to a transfer machine. IEEE Trans Ind Electron 2007; 54(1): 638–650.

Yagi

Harada

Kobayashi

. Upper-limb power-assist control for agriculture load lifting. Int J Autom Technol 2009; 3(6): 716–722.

Dimeas

Koustoumpardis

Aspragathos

. Admittance neuro-control of a lifting device to reduce human effort. Adv Robot 2013; 27(13): 1013–1022.

Hara

Sankai

. HAL equipped with passive mechanism. In: Proceedings of IEEE international symposium on system integration, 2012, pp. 1–6.

Gosselin

Laliberte

Mayer-St-Onge

. A friendly beast of burden: a human-assistive robot for handling large payloads. IEEE Robot Autom Mag 2013; 20(4): 139–147.

Lecours

Mayer-St-Onge

Gosselin

. Variable admittance control of a four-degree-of-freedom intelligent assist device. In: Proceedings of IEEE international conference on robotics and automation, 2012, pp. 3903–3908.

10.

Colgate

Peshkin

Klostermeyer

. Intelligent assist devices in industrial applications: a review. In: Proceedings of IEEE/RSJ international conference on intelligent robots and systems, Vol. 3, 2003, pp. 2516–2521.

11.

Olivier

Sylvain

Fares

. Cobomanip: a new generation of intelligent assist device. In: Proceedings of 41st international symposium on robotics, 2014, pp. 1–8.

12.

Rahman

Ikeura

. Weight-perception-based novel control of a power-assist robot for the cooperative lifting of light-weight objects. Int J Adv Robot Syst 2012; 9(118): 1–13.

13.

http://www.rb3d.com/en/cobots-range/ (accessed 08 March 2018).

14.

Rahman

Ikeura

Nobe

. Weight-perception-based model of power assist system for lifting objects. Int J Autom Technol 2009; 3(6): 681–691.

15.

Rahman

Ikeura

Nobe

. Unimanual and bimanual weight discrimination in lifting objects with a power assist system. In: Proceedings of ICROS-SICE international joint conference, 2009, pp. 4787–4792.

16.

Rahman

Ikeura

Nobe

. A psychophysical model of the power assist system for lifting objects. In: Proceedings of IEEE international conference on systems, man, and cybernetics, 2009, pp. 4125–4130.

17.

Rahman

Wang

Walker

. Trust-based compliant robot-human handovers of payloads in collaborative assembly in flexible manufacturing. In: Proceedings of the 12th IEEE international conference on automation science and engineering (IEEE CASE 2016), Texas, USA, 21–24 August 2016, pp. 355–360.

18.

Rahman

Ikeura

. Cognition-based control and optimization algorithms for optimizing human-robot interactions in power assisted object manipulation. J Inform Sci Eng 2016; 32(5): 1325–1344.

19.

Rahman

Ikeura

. Weight-prediction-based predictive optimal position and force controls of a power assist robotic system for object manipulation. IEEE Trans Ind Electron 2016; 63(9): 5964–5975.

20.

Rahman

Ikeura

Hayakawa

. Novel human-centric force control methods of power assist robots for object manipulation. In: Proceedings of 2013 IEEE international conference on robotics and biomimetics (IEEE ROBIO 2013), pp. 340–345.

21.

Rahman

Ikeura

. Optimizing perceived heaviness and motion for lifting objects with a power assist robot system considering change in time constant. Int J Smart Sens Intell Syst 2012; 5(2): 458–486.

22.

Duchaine

Gosselin

. Safe, stable and intuitive control for physical human-robot interaction. In: Proceedings of IEEE international conference on robotics and automation, 2009, pp. 3383–3388.

23.

Rahman

Ikeura

. Improving interactions between a power assist robot system and its human user in horizontal transfer of objects using a novel adaptive control method. Adv Human Comput Int 2012; 2012: 12. Article ID 745216.

24.

Rahman

Ikeura

Sawai

. Worst-cases prediction by human in lifting objects with a power assist robot system: effectiveness of a novel control strategy to improve the system performance in worst-cases. In: Proceedings of 2010 IEEE international conference on biomedical robotics and biomechatronics, Tokyo, 26–29 September 2010, pp. 136–142.

25.

Rahman

Ikeura

Hayakawa

. Psychophysical relationships between actual and perceived weights for lifting objects with power-assist: consideration of constrained and unconstrained lifting. In: Proceedings of 2010 IEEE/SICE international symposium on system integration, Japan, 21–22 December 2010, pp. 152–157.

26.

Romat

Williams

Wang

. Natural human-robot interaction using social cues. In: Proceedings of ACM/IEEE international conference on human-robot interaction, 2016, pp. 503–504.

27.

Rahman

. Evaluating and benchmarking the interactions between a humanoid robot and a virtual human for a real-world social task. In: Proceedings of the 6th international conference on advances in information technology (IAIT 2013), communications in computer and information science, Bangkok, Thailand, 12–13 December 2013, Vol. 409, pp. 184–197. Springer.

28.

Woods

Walters

Koay

. Comparing human robot interaction scenarios using live and video based methods: towards a novel methodological approach. In: Proceedings of 9th IEEE international workshop on advanced motion control, Istanbul, 2006, pp. 750–755.

29.

Rahman

Ikeura

. MPC to optimise performance in power-assisted manipulation of industrial objects. IET Electric Power Appl 2017; 11(7): 1235–1244.

30.

Igarashi

Katsura

. Position based free-motion data connecting by using minimum force-differential model. In: Proceedings of IEEE international conference on mechatronics, 2015, pp. 535–540.

31.

Rahman

Wang

. Dynamic affection-based motion control of a humanoid robot to collaborate with human in flexible assembly in manufacturing. In: Proceedings of ASME dynamic systems and controls conference, Columbus, Ohio, 28–30 October 2015, Paper No. DSCC2015-9841, pp. V003T40A005.

32.

Rahman

. Cyber-physical-social system between a humanoid robot and a virtual human through a shared platform for adaptive agent ecology. IEEE/CAA J Autom Sin 2018; 5(1): 190–203.

33.

https://www.mathworks.com/help/gads/what-is-global-optimization.html (accessed 08 March 2018).

34.

Rao

Classical optimization techniques, Ch.2. In: Engineering Optimization: Theory and Practice. Hoboken, NJ, USA: John Wiley & Sons, Inc., 2009.

35.

Rahman

Sadr

Wang

. Trust-based optimal subtask allocation and model predictive control for human-robot collaborative assembly in manufacturing. In: Proceedings of ASME dynamic systems and controls conference, Ohio, 28–30 October 2015, Paper No. DSCC2015-9850, pp. V002T32A004, p. 10.

36.

Rawlings

Mayne

. Model predictive control: theory and design, 5th ed. Nob Hill Publishing, ISBN 9780975937709, 2015.

37.

Kouvaritakis

Cannon

. Model predictive control: classical, robust and stochastic. advanced textbooks in control and signal processing, 1st ed. Springer, ISBN-13 978-3319248516, 2016.

38.

Minamiyama

Kiyota

Mori

. Development of constant torque device and its application to power assist systems. In: Proceedings of IEEE international conference on advanced intelligent mechatronics, 2014, pp. 192–197.

39.

Yamada

Hasuike

. Document image processing based on enhanced border following algorithm. In: Proceedings of the 10th international conference on pattern recognition, 1990, pp. 231–236.

40.

Santoso

Garratt

Anavatti

. Visual–inertial navigation systems for aerial robotics: sensor fusion and technology. IEEE Trans Autom Sci Eng 2017; 14(1): 260–275.

41.

Rahman

Ikeura

Nobe

. Study on optimum maneuverability in horizontal manipulation of objects with power-assist based on weight perception. In: Proceedings of SPIE, Vol. 7500, 75000P, 2010.

42.

De Carli

Davide

Hohert

Evan

Parker

Chris AC

. Measuring intent in human-robot cooperative manipulation. In: Proceedings of IEEE international workshop on haptic audio visual environments and games, 2009, pp. 159–163.

43.

Andò

Ascia

. Navigation aids for the visually impaired: from artificial codification to natural sensing. IEEE Instrumentat Meas Mag 2007; 10(3): 44–51.