Sage Journals: Discover world-class research

Abstract

Teleoperation has become increasingly important for the real-world application of robots. However, in current teleoperation applications, operators need to visually recognize any physical deviations between the robot and themselves and correct its operation accordingly. Even when humans move their bodies, achieving high positional accuracy can be difficult for them, which can limit their speed of movement and place a heavy burden on them. In this study, we proposed a parameter optimization method for feedforward compensation of the link-length deviation and operator's habitual body movements in leader–follower control in 3D space. To optimize the parameters, we used Digital Annealer developed by Fujitsu Ltd, which could rapidly solve the combinatorial optimization problem. The objective function minimized the difference between the hand positions of the robot and targets. Simulations verified that the proposed method could reduce hand positional differences.

Keywords

Robot teleoperation optimization quantum computing

Introduction

Over the past few decades, the demand for robotic teleoperation applications in various sectors has increased, from industry to service robotics.^1–3 Robotic teleoperation can be useful in complicated environments—including those with humans—because it can be difficult for robots to recognize the environment adequately, avoid collisions with furniture and humans, and automatically perform tasks.

Several studies have been conducted to improve robotic teleoperation. For example, controllers have been used to measure the target values of robots,^4–7 some methods calculating joint angles by performing inverse kinematic calculations on the robot using the positional and posture information of a hand-held controller to manipulate the robot.^8–10 These studies focused on the operability of the hand position, which was an important functional approach and was easy to implement. Wang et al. investigated the reaction times between joint-space and task-space commands using an exoskeletal motion-capture device. They confirmed that most subjects adapted to task-space mapping faster than to joint-space mapping after similar levels of practice.¹¹ However, because only the hand position and posture could be controlled, it was impossible to move only the elbow position to avoid obstacles in front of the target. To improve operability, it was necessary to consider the movements of the operator links and joints. Consequently, leader–follower controllers—known as master–slave controllers—have been used in various teleoperation applications. Here, the link positions, orientations, and joint angles of operators are measured using a motion-capture system or an inertial-measurement unit, the measured motions being used as robotic commands.^12–15

However, in a conventional leader–follower controller, the operator must pay careful attention to the robot's movement, visually recognizing the robot's movement and modifying it sequentially. This can cause operator fatigue. Additionally, the speed of motion cannot simply be increased because of the delay in the robot's motion (as directed by operator) and the delay in the camera image used by the operator to recognize the robot's motion.

Owing to two types of gaps, an operator may have to observe the robot's movements. One is the gap in the number of joints, size, and speed of movement between the robot and the operator. If a robot is of similar size and has the same number of joints as an operator, the operator can easily control it. However, a humanoid robot of similar size and with the same number of joints as an operator can be extremely complex and expensive, making it difficult to introduce such a robot into the real world. The second gap is that between the operator's body image and the actual body when the operator moves their own body. Even when an operator moves their body, achieving high positional accuracy can be difficult without visual feedback. For example, when operators move their arms 30° to the right from the front without visual feedback, their arm tending to shift by a few degrees. In this study, we refer to this gap between the operator's imaginary position and the actual position of the operator's fingertip as a habitual movement.

In practice, only structural differences have been studied, and the compensation for habitual movements has not been studied. To compensate for the difference in the number of joints between an operator and a robot, a matching technique that reflects only the movements of the dominant joints in the movements of the robot joints has been used.¹⁶ This method has also been employed in the operation of virtual reality characters.¹⁷ Furthermore, if the link size of the operator and robot differ, extending an arm to a certain target can cause the operator and robot to adopt different hand positions, reducing operability. Nakamura et al. proposed a self-body image update method for augmented reality and investigated how operators learned body-link lengths.¹⁸ Regarding the difference in the size of an operator and a robot, a method for measuring the link size of the operator in advance and reflecting it at the time of operation has been reported. However, considerable time and effort were required to perform these measurements in advance.¹⁹

Some researchers have proposed a parameter optimization method for correcting differences in the link length of a robot and an operator for feedforward compensation in a leader–follower controller that synchronizes the robot with the movement of the operator.²⁰ To correct for the differences in link length between the operator and 2-joint 3-link serial robots, the shoulder and elbow joint angles were measured when the operators stretched their own hands to a predetermined target hand position. A target function was set to minimize the difference between these hand positions, such that the hand position normalized using the total arm length of the robot was equal to that normalized using the total arm length of the operator. After converting the minimization problem to a quadratic unconstrained binary optimization (QUBO) problem, optimization calculations could be performed.

However, when teleoperating a commonly used serial-link robot to optimize the parameters established for each joint angle, the hand position can involve a complex equation (potentially exceeding the third order) as the number of joints increases, making optimization using the QUBO method difficult. Additionally, because it can only be optimized for one type of preliminary motion in a 2D plane, robotic teleoperation requiring positional accuracy at multiple hand positions—such as the initial position of the object and the location of the object when the robot grasps it and places it in another location—is not possible. However, if optimization can be performed by simultaneously considering the preliminary movements of multiple target hand positions, the robot's hand positions will be closer to those intended by the operator. Furthermore, because the proposed method matches the target hand position with the operator's hand position when the operator performs a preliminary movement, even if the operator's hand position differs from that of the target owing to the operator's habitual movements, the target hand position can be achieved if the operator performs a habitual movement. If the proposed method could provide parameters that automatically compensate for such habitual motion, the load on the operator can be reduced.

In this study, we propose an optimization method for compensation parameters that decreases the influence of the operator's habitual movements and misalignment of the link length between the robot and operator when the robot is controlled in 3D space. Specifically, the compensation parameters were obtained by solving an optimization problem to minimize the deviation of the robot's hand position from the target hand position. By reflecting the optimized set of compensation parameters in real-time, the robot could be controlled as expected through natural motion without requiring visual feedback, thereby reducing the operator's mental burden and need for practice during operation (Figure 1).

Figure 1.

Schematic view of the proposed method. With the compensation parameters optimized by preliminary operations, the proposed method decreased the gap of the robot hand position to the target. The gaps are influenced by link length differences between an operator and a robot, and habitual movements—that is, the displacement between the operator's imaginary position and the actual position of the operator's fingertip.

Methods

Optimization framework to compensate for habitual movements

The proposed method comprises the following steps:

Preliminary operations for measuring habitual movement by comparing the predetermined target position and operator hand position.

Compensation parameter optimization to minimize the impact of habitual movements.

Teleoperation using the compensation parameters.

Compensation parameters multiplied by the hand position which are normalized by the operator's arm length was formulated. And the compensation parameters are used with the normalized hand position command value of the robot. To determine the compensation parameters before teleoperation, the operator performed a preliminary operation to create a dataset for the operator-normalized hand position relative to the operator's target-normalized hand position. Subsequently, the minimization problem of the difference between the operator's target-normalized hand position and the robot's target-normalized hand position could be solved to ensure that a parameter search was performed. This parameter mapped the operator-normalized hand position to the robot-normalized hand position in the 3D space.

Consider multiple targets—that is, hand movements—to connect each target position. For example, if two targets are considered, mapping can be performed to map a hand movement that connects two points using a straight line. When transforming multiple hand positions, it is necessary to compensate for the fact that the distance between the hand positions of the operator normalized by the arm length of the operator may differ from that of the target position. Additionally, even if the distance between multiple positions is accurate, if any hand position deviates from the target, it is necessary to shift the starting point of the hand movement to match this deviation. Consequently, the transformation equation in the X-direction can be expressed as follows:

x_{c o m p} = x_{o p e r a t o r} * k_{x} + d_{x}

(1)

where

x_{c o m p}

denotes the robot's target-normalized hand position in the X-direction,

x_{o p e r a t o r}

denotes the operator-normalized hand position,

k_{x}

denotes a parameter for multiple hand positions, and

d_{x}

denotes the offset parameter.

In this study, the X-direction is referred to as the left/right direction, the Y-direction as the front/back direction, and the Z-direction as the up/down direction. Equation (1) can be used to perform independent calculations in each of the three directions of the robot's target-normalized hand position to correct the positions in each direction in the 3-D space, expressed as follows:

y_{c o m p} = y_{o p e r a t o r} * k_{y} + d_{y}

(2)

z_{c o m p} = z_{o p e r a t o r} * k_{z} + d_{z}

(3)

Implementation of Digital Annealer

Various parameter optimization methods have been proposed. The problem addressed in this study was the optimization of a combinatorial problem of the gain and offset parameters established to calculate a robot's target hand position in 3D space. Because it is comparatively fast and less likely to fall into local minimums, we used a new technology—that is, Digital Annealer, developed by Fujitsu Ltd—which could solve the combinatorial optimization problem quickly using a computational method that mimicked quantum annealing.²¹ Using a digital circuit design inspired by quantum phenomena, Digital Annealer focuses on rapidly solving complex combinatorial optimization problems without the added complications and costs typically associated with quantum computing methods. This enables faster computation of QUBO than conventional methods, such as simulated annealing.²² To describe the combinatorial optimization problem, the QUBO model can be expressed as follows:

E_{x} = \sum_{i = 1}^{n} \sum_{j = 1}^{i} c_{i j} x_{i} x_{j}

(4)

where

c_{i j}

denotes a coefficient and x denotes a binary.

In this study, the error between the target value of the hand position and the current value of the robot was minimized because the compensation parameter was set within a finite range. To achieve this, optimized compensation parameters were selected from this range. Moreover, the QUBO model was developed as a simplified model to handle compensation parameters based on the objective function and constraints. The objective function ( $H_{o b j e c t i v e}$ ) can be expressed as follows:

H_{o b j e c t i v e} = (q_{t a r g e t} - q_{c o m p})^{2},

(5)

where

q_{t a r g e t}

denotes the target hand position and

q_{c o m p} (x_{c o m p}, y_{c o m p}, z_{c o m p})

denotes the hand position of the robot with compensation parameters.

In equation (5), each gain and offset parameter can be discretized. In this study, the coordinates were set at the shoulder joint. To consider hand movements that have positive or negative changes, the hand position of the operator can be offset by the arm length of the operator, which can measure the hand position during arm stretching. In this study, the hand positions in the X- and Z-directions were offset. Therefore, equations (1)–(3) can be rewritten and expressed as follows:

x_{c o m p} = (x_{o p e r a t o r} + L_{o p e r a t o r}) * k_{x_{d i s c r e t i z e d}} * r_{g a i n} - L_{o p e r a t o r} + d_{x_d i s c r e t i z e d} * r_{o f f s e t}

(6)

y_{c o m p} = y_{o p e r a t o r} * k_{y_{d i s c r e t i z e d}} * r_{g a i n} + d_{y_d i s c r e t i z e d} * r_{o f f s e t}

(7)

z_{c o m p} = (z_{o p e r a t o r} + L_{o p e r a t o r}) * k_{y_d i s c r e t i z e d} * r_{g a i n} - L_{o p e r a t o r} + d_{z_d i s c r e t i z e d} * r_{o f f s e t}

(8)

where

r_{g a i n}

denotes the resolution of the discretized gain parameters,

k_{x_d i s c r e t i z e d}

k_{y_d i s c r e t i z e d}

, and

k_{z_d i s c r e t i z e d}

denote the discretized gain parameters,

r_{o f f s e t}

denotes the resolution of the discretized offset parameters, and

d_{x_d i s c r e t i z e d}

d_{y_d i s c r e t i z e d}

, and

d_{z_d i s c r e t i z e d}

denote the discretized offset parameters. Moreover, a constraint function (

H_{c o n s t r a i n t}

) should be included, expressed as follows:

H_{c o n s t r a i n t_g a i n} = \sum_{j}^{d i r e c t i o n_n u m} {(1 - \sum_{h}^{r a n g e} k_{j h})}^{2}

(9)

H_{c o n s t r a i n t_o f f s e t} = \sum_{j}^{d i r e c t i o n_n u m} {(1 - \sum_{h}^{r a n g e} d_{j h})}^{2}

(10)

The constraint function indicates that only one compensation parameter exists for each gain and offset value. The Hamiltonian energy function (H) should be constructed from the objective and constraint functions, expressed as follows:

H = H_{o b j e c t i v e} + w * H_{c o n s t r a i n t},

(11)

where w denotes the weighting factor between the objective and constraint functions.

The translation from the optimization problem to the QUBO model comprises the following steps:

Formulate the energy function.

Encode the integer variables with binary variables.

Obtain the QUBO matrix from the coefficients.

Programming QUBO models for Ising machines can be difficult if the energy function is complicated, as equation (11) must be translated. To construct the QUBO model using equation (11), we used PyQUBO as the Python library.²³ Using this library, programming could be practically identical to the shape of the equation constructed using Steps (2) and (3). Consequently, users only construct an objective function. Digital Annealer was then used repeatedly for several iterations to minimize the energy of the model, solve the problem using the QUBO model, and determine the compensation parameters.

Experimental results

Simulation conditions

In the simulations, compensation parameter searches using Digital Annealer were performed on the prepared datasets at two points as a preliminary operation. The parameters used for Digital Annealer are summarized in Table 1.

Table 1.

Digital Annealer simulation parameters.

Parameter (unit)	Value
Robot upper arm length (mm)	300
Robot forearm length (mm)	150
Resolution of gain r_gain	0.05
Resolution of offset r_offset (mm)	5
Max. number of discretization	40
Weighting factor w	100,000
Number of iterations	1,000,000

The number of iterations affects the time required to find the globally optimal solution without falling into a locally optimal solution. In the simulation, the number of iterations was set to 1,000,000 to find the optimal global solution. Moreover, Digital Annealer can increase the probability of obtaining an optimum solution. If no bit-flip candidate is found, escape from a local minimum state is facilitated by adding a positive offset to the energy.²¹ To decrease the calculation time, the number of iterations and implementations in Digital Annealer must be increased. Consequently, the robot target-normalized hand position calculated using the obtained compensation parameters was then applied to the robot model, and each joint angle of the robot was calculated and driven using inverse kinematic calculations.

Link size compensation simulation

Through simulations, we verified whether the proposed method could be used for 3D spatial teleoperation fitting that did not require prior measurements of each link length of the operator. For example, if the operator moved their hand to the target-normalized hand position, similar to the preliminary operations, two sets of motion data were generated from the starting point to the goal point of the motion as both preliminary motions and teleoperation tasks. Optimization calculations were then performed for each motion. Simplified arm models of an operator and a robot were constructed, the average length of the operator's upper arm and forearm being 300 and 240 mm, respectively, the forearm length being approximately 80% that of the upper arm.²⁴

The robot arms were set to 300 mm for the upper arm and 150 mm for the forearm so that the link length ratio of the upper arm to the forearm was different from that of the operators. In parameter optimization, the operator hand position data in the prepared dataset were normalized using the total arm length calculated from the hand position when the operator extended their arm straight, and the operator-normalized hand position was used.

The experimental results summarized in Tables 2 and 3 show the obtained compensation parameters, target hand positions, robotic hand positions without compensation, and robotic hand positions when the compensation parameters were used for each dataset. The results of each simulation are shown in Figure 2. Under all experimental conditions, the robotic hand position is several cm away from the target without compensation; however, with compensation, the gap is reduced to approximately one cm. It is evident that the parameters obtained using the proposed method can be used for teleoperation at both target points with a high hand positional accuracy.

Figure 2.

Simulation results of the link size compensation experiment. In each simulation, the left-hand graph shows the result for the start point, the right-hand graph showing the result for the goal point. The yellow point indicates the target position of the robot hand. The green links indicate an operator arm. The blue links indicate a robotic arm without compensation. The red links indicate a robotic arm with compensation.

Table 2.

Obtained compensation parameters.

	Compensation parameters	X	Y	Z
Motion A	Gain	0.8	1.05	1.0
Motion A	Offset	75	0	0
Motion B	Gain	0.8	1.0	1.1
Motion B	Offset	80	0	−60

Table 3.

Results of simulations with and without compensation.

	Motion A						Motion B
	Start			Goal			Start			Goal
	X	Y	Z	X	Y	Z	X	Y	Z	X	Y	Z
Ratio of target hand position to total arm length (%)	20	30	10	20	70	10	20	20	40	20	70	40
Operator hand position (mm)	100	150	50	100	350	50	100	100	200	100	350	200
Target hand position of the robot (mm)	90	135	45	90	315	45	90	90	180	90	315	180
Robot hand position without compensation (mm)	147	133	44	132	311	44	143	89	178	124	312	178
Robot hand position with compensation (mm)	102	140	44	90	327	44	104	89	180	89	312	181
Distance between the hand position of the target and robot without compensation (mm)	57.0			42.2			53.0			34.2
Distance between the hand position of the target and robot with compensation (mm)	13.0			12.0			14.0			3.3

Simulations to compensate for the habitual movements of an operator in 3D space

We verified, through simulations, that the proposed method could automatically compensate for inaccuracies in hand position caused by the operator's own habitual movements when moving their own body to control the robot. As in the previous experiment, we prepared the target motion and the operator's hand position coordinate data assuming that the operator moved their own hand in relation to the target-normalized hand position.

The optimization calculations were performed for each of five sets—that is, the operator-normalized hand position did not deviate from the target-normalized hand position at either of the two points; a deviation of 5% occurred to the right at both points; both points were 10% off to the right; one of the two points was 5% off to the right and the other was 5% off to the left, and one of the two points was 10% off to the right and the other was 10% off to the left. The operator model, robot model, and experimental methods were the same as those used in the previous experiments.

The compensation parameters and experimental results obtained are listed in Tables 4 and 5, respectively. The results of each simulation are shown in Figure 3.

Figure 3.

Simulation results of compensation for habitual movements in the horizontal plane. In each simulation, the left-hand graph shows the results for the start point and the right-hand graph shows the results for the goal point. The yellow point indicates the target position of the robotic hand. The green links indicate an operator arm. The blue links indicate a robotic arm without compensation. The red links indicate a robotic arm with compensation.

Table 4.

Obtained compensation parameters for habitual movements.

	Compensation parameters	X	Y	Z
(a) No deviation	Gain	0.75	0.95	0.85
(a) No deviation	Offset	100	5	75
(b) Deviation of 10% to the right	Gain	0.7	0.95	1.15
(b) Deviation of 10% to the right	Offset	100	25	−70
(c) Deviation of 20% to the right	Gain	0.8	0.95	0.85
(c) Deviation of 20% to the right	Offset	40	30	80
(d) Deviation of 10% to the right and left	Gain	0.75	0.8	1.0
(d) Deviation of 10% to the right and left	Offset	30	70	30
(e) Deviation of 20% to the right and left	Gain	0.75	0.65	0.9
(e) Deviation of 20% to the right and left	Offset	100	65	50

Table 5.

Simulation results of compensation for habitual movements.

	(a) No deviation						(b) Deviation of 10% to the right						(c) Deviation of 20% to the right						(d) Deviation of 10% to the right and center						(e) Deviation of 20% to the right and center
	Start			Goal			Start			Goal			Start			Goal			Start			Goal			Start			Goal
	X	Y	Z	X	Y	Z	X	Y	Z	X	Y	Z	X	Y	Z	X	Y	Z	X	Y	Z	X	Y	Z	X	Y	Z	X	Y	Z
Ratio of target hand position to total arm length (%)	20	30	10	20	50	10	20	30	10	20	50	10	20	30	10	20	50	10	20	30	10	20	50	10	20	30	10	20	50	10
Ratio of movement habits to total arm length (%)	0	0	0	0	0	0	10	0	0	10	0	0	20	0	0	20	0	0	10	0	0	−10	0	0	20	0	0	−20	0	0
Operator hand position (mm)	100	150	50	100	250	50	150	150	50	150	250	50	200	150	50	200	250	50	150	150	50	50	250	50	200	150	50	0	250	50
Operator movement habits (mm)	0	0	0	0	0	0	50	0	0	50	0	0	100	0	0	100	0	0	50	0	0	−50	0	0	100	0	0	−100	0	0
Target hand position of the robot (mm)	90	135	45	90	225	45	90	135	45	90	225	45	90	135	45	90	225	45	90	135	45	90	225	45	90	135	45	90	225	45
Robot hand position without compensation (mm)	147	133	44	141	222	44	187	118	39	183	213	43	225	109	36	223	206	41	187	118	39	97	233	47	225	109	36	50	243	49
Robot hand position with compensation (mm)	102	140	44	93	216	45	96	137	43	93	227	46	96	133	43	94	226	48	128	135	39	60	226	47	156	136	38	25	223	49
Distance between the hand position of the target and robot without compensation (mm)	57.0			51.1			98.7			93.8			137.8			134.4			98.7			10.8			137.8			44.0
Distance between the hand position of the target and robot with compensation (mm)	13.0			9.5			6.6			3.7			6.6			5.1			38.5			30.1			66.4			65.2

Experiments to compensate for the habitual movements of operators

We verified that the proposed method could compensate for inaccuracies in hand position caused by the operator's own habitual movements. The motions of three subjects were measured using Microsoft Azure Kinect as a motion capture. Optimization calculations were performed based on average of the measured data with respect to the motion targets, and the robot command values were obtained by adapting the obtained parameters to the operator's motions. The robot model and experimental methods were the same as those used in the previous simulations.

The compensation parameters and experimental results obtained are listed in Tables 6 and 7, respectively. The experimental results of one of the subjects are shown in Figure 4. As indicated by the displacement of the operator motion relative to the target position, the operator motion does not always reach the position desired by the operator in the brain because the operator motion involves habitual movements. Thus, the motion of the robot without compensation, which only uses the operator motion data, has a large displacement from the robot's target. On the other hand, the compensated robot motion is confirmed to be closer to the robot target.

Figure 4.

Experimental results of compensation for habitual movements in the horizontal plane.

Table 6.

Obtained compensation parameters for habitual movements.

	Compensation parameters	X	Y	Z
Subject A	Gain	0.70	0.95	0.8
Subject A	Offset	80	95	95
Subjec B	Gain	0.6	1.3	0.8
Subjec B	Offset	100	25	75
Subjec C	Gain	0.85	1.6	1.15
Subjec C	Offset	−20	75	−50

Table 7.

Results of experiments with and without compensation.

	Subject A						Subject B						Subject C
	Start			Goal			Start			Goal			Start			Goal
	X	Y	Z	X	Y	Z	X	Y	Z	X	Y	Z	X	Y	Z	X	Y	Z
Ratio of target hand position to total arm length (%)	20	30	0	20	50	0	20	30	0	20	50	0	20	30	0	20	50	0
Operator hand position (mm)	129	141	14	147	247	27	255	210	54	320	283	32	141	183	−31	130	196	−32
Target hand position of the robot (mm)	90	135	0	90	225	0	90	135	0	90	225	0	90	135	0	90	225	0
Robot hand position without compensation (mm)	224	46	3	228	144	15	273	85	22	311	154	17	213	76	−13	208	94	−15
Robot hand position with compensation (mm)	102	136	28	105	225	−12	79	132	−21	101	218	−24	92	159	−20	89	193	−22
Distance between the hand position of the target and robot without compensation (mm)	161			160.7			190.0			232.4			137.8			134.4
Distance between the hand position of the target and robot with compensation (mm)	30.5			20.1			24.1			27.9			31.4			39.2

Discussion

The proposed method could bring the robot's hand position closer to the target, even when the operator's hand position was incorrect relative to the target because of the operator's movements. However, if the direction of habitual movement was different for the target and the operator's hand positions at the two points, some positional errors remained, as the compensation parameters used in this study were gain and offset. Coordinate transformations using gain and offset values are part of the affine and homographic transformations frequently used in coordinate transformations. Moreover, a low number of parameter terms does not allow for deformations (such as rotation)—consequently, when two targets were used in this simulation, compensating for the slope of the line connecting the two target points and the line connecting the two robotic hand positions was not possible. Coordinate transformations that are more complex (including rotational and shear deformation) can be achieved by adding more parameters to the optimization calculations.

Various optimization, reinforcement learning, and deep learning techniques have been used in teleoperation to achieve high operability.^25–27 In this study, we mainly focused on compensating for habitual movements, not the optimization techniques. Therefore, a comparison of the calculation overhead and accuracy with other methods could contribute to the field of robotic teleoperation applications.

Conclusions

Two main contributions of this study are as follows:

First, we proposed a parameter optimization method that automatically compensated for the differences in the link size and structure of an operator and robot when teleoperating a robot in 3D space. With existing leader–follower controllers, there have been cases in which the robot could not be controlled as intended by the operator owing to differences in the size and number of joints of the robot and operator. A previous study by the authors was limited to the horizontal plane and used a method to set compensation gains for the joint angles of a 2-joint robot. Hence, it could not be used to control an articulated robot in 3D space. However, the method proposed in this study did not optimize the joint angle. Instead, it optimized the hand positions in each direction in 3D space using compensation gains and offsets. Hence, it could be used in 3D space. Furthermore, differences in the robot structure could be compensated for by providing the robot with an optimized robot target position, performing inverse kinematics calculations within the robot. Consequently, the proposed method could be used regardless of the number of joints in the robot.

Second, the leader–follower controller automatically compensated for the operator's habitual movements when controlling the robot. The proposed method optimized the pose assumed by the operator while performing preliminary movements via teleoperation. Therefore, the operator was not required to hold the exact pose the robot was required to hold, enabling the operator to control the robot easily by considering their own physical operational habits. Consequently, a high degree of operability could be maintained, such as when operating a robot in an environment with obstacles and reaching a position that was blocked by an obstacle and was not visible to the operator. Since the operator does not need to recognize and adjust the position of the robot hand, it is expected to be easier to control the robot in environments with long visual delays, such as remote control in outer space. Moreover, the proposed method would be useful for teleoperation for reaching to objects which move fast because the proposed teleoperation does not need the visual feedback for an operator.

In the future, we aim to compare the calculation overhead and accuracy of the proposed optimization method with those of other methods and verify the operability and operator burden using the proposed method.

Footnotes

Acknowledgments

This study was conducted with the support of the Research Institute for Science and Engineering, Waseda University; the Human Performance Laboratory, Waseda University; the Future Robotics Organization, Waseda University; and as part of the humanoid project at the Humanoid Robotics Institute, Waseda University. It was also financially supported in part by the JSPS KAKENHI Grant No. 21K17832. We thank all of these for the financial and technical support provided.

Data availability

All data analyzed in this study have been included in the published article. Specific datasets are available from the corresponding author upon request.

Declaration of conflicting interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the Japan Society for the Promotion of Science (grant number 21K17832).

ORCID iD

Takuya Otani

References

Liu

Habibnezhad

Jebelli

. Brain-computer interface for hands-free teleoperation of construction robots. Autom Constr 2021; 123: 103523.

Gonzalez

Agarwal

Balakuntala

, et al. DESERTS: DElay-tolerant SEmi-autonomous Robot Teleoperation for Surgery. In: Proceedings of the 2021 IEEE international conference on robotics and automation, Xi'an, China: IEEE, 30 May–5 June 2021, pp.12693–12700.

Young

Peschel

. Review of human–machine interfaces for small unmanned systems with robotic manipulators. IEEE Trans Human-Mach Syst 2020; 50: 131–143.

Sian

Yokoi

Kajita

, et al. Whole body teleoperation of a humanoid robot integrating operator's intention and robot's autonomy: an experimental verification. In: Proceedings 2003 IEEE/RSJ international conference on intelligent robots and systems, Las Vegas, NV, USA: IEEE, 27–31 Oct 2003, pp.1651–1656.

Shi

Liu

Mei

. Accurate parameter estimation for master–slave operation of a surgical robot. Machines 2021; 9: 1–18.

Gonzalez

Solanes

Munoz

, et al. Advanced teleoperation and control system for industrial robots based on augmented virtuality and haptic feedback. J Manuf Syst 2021; 59: 283–298.

Kong

Pang

, et al. Gulim: a hybrid motion mapping technique for teleoperation of medical assistive robot in combating the COVID-19 pandemic. IEEE Trans Med Robot Bionics 2022; 4: 106–117.

Zhang

McCarthy

Jow

, et al. Deep imitation learning for complex manipulation tasks from virtual reality teleoperation. In: Proceedings of the 2018 IEEE international conference on robotics and automation, Brisbane, QLD, Australia: IEEE, 21–25 May 2018, pp.5628–5635.

Deng

YWW

, et al. An intelligent interaction framework for teleoperation based on human-machine cooperation. IEEE Trans Human-Mach Syst 2022; 52: 963–972.

10.

Laghi

Raiano

Amadio

, et al. A target-guided telemanipulation architecture for assisted grasping. IEEE Robot Autom Lett 2022; 7: 8759–8766.

11.

Wang

Murphy

Kenney

, et al. A comparison between joint space and task space mappings for dynamic teleoperation of an anthropomorphic robotic arm in reaction tests. In: Proceedings of the 2021 IEEE international conference on robotics and automation, Xi'an, China: IEEE, 30 May–5 June 2021, pp.2846–2852.

12.

LNM

Siles

. Teleoperation of a humanoid robot using an optical motion capture system. In: Proceedings of the 2018 IEEE international work conference on bioinspired intelligence, San Carlos, Costa Rica: IEEE, 18–20 July 2018, pp.1–8.

13.

Miller

Jenkins

Kallmann

, et al. Motion capture from inertial sensing for untethered humanoid teleoperation. In: Proceedings of the IEEE-RAS international conference on humanoid robotics, Santa Monica, CA, USA: IEEE, 10–12 Nov 2004, pp.547–565.

14.

Cao

Hidalgo

Simon

, et al. Openpose: realtime multi-person 2D pose estimation using part affinity fields. IEEE Trans Pattern Anal Mach Intell 2021; 43: 172–186.

15.

Digo

Antonelli

Cornagliotto

, et al. Collection and analysis of human upper limbs motion features for collaborative robotic applications. Robotics 2020; 9: 1–12.

16.

Enayati

Vantadori

, et al. Online human-like redundancy optimization for tele-operated anthropomorphic manipulators. Int J Adv Robot Syst 2018; 15: 1–13.

17.

Gleicher

. Retargetting motion to new characters. In: Proceedings of the 25th annual conference on computer graphics and interactive techniques. New York, NY, USA: Association for Computing Machinery, 19–24 July 1998, pp.33–42.

18.

Nakamura

Mochizuki

Konno

, et al. Research on updating of body schema using AR limb and measurement of the updated value. IEEE Syst J 2016; 10: 903–911.

19.

Penco

Clement

Modugno

, et al. Robust real-time whole-body motion retargeting from human to humanoid. In: Proceedings of the IEEE-RAS international conference on humanoid robotics, Beijing, China: IEEE, 6–9 Nov 2018, pp.425–432.

20.

Otani

Takanishi

Nakamura

, et al. Optimization of link length fitting between an operator and a robot with Digital Annealer for a leader-follower operation. Robotics 2022; 11: 12.

21.

Matsubara

Takatsu

Miyazawa

, et al. Digital Annealer for high-speed solving of combinatorial optimization problems and its applications. In: Proceedings of 2020 25th Asia and South Pacific design automation conference (ASP-DAC). Beijing, China: IEEE, 13–16 Jan 2020, pp.667–672.

22.

Sao

Watanabe

Musha

, et al. Application of Digital Annealer for faster combinatorial optimization. Fujitsu Sci Tech J 2019; 55: 45–51.

23.

Zaman

Tanahashi

Tanaka

. PyQUBO: python library for mapping combinatorial optimization problems to QUBO form. IEEE Trans Comput 2022; 71: 838–850.

24.

Kouchi

Mochimaru

. Japanese 3-D body shape and dimensions data 2003. Japan: National Institute of Advanced Industrial Science and Technology, 2005.

25.

Gomes

Radhakrishnan

Penco

, et al. Humanoid whole-body movement optimization from retargeted human motions. In: Proceedings of the 2019 IEEE-RAS 19th international conference on humanoid robots, Toronto, ON, Canada: IEEE, 15–17 Oct 2019, pp.178–185.

26.

Guzman

Morellas

Papanikolopoulos

. Robotic embodiment of human-like motor skills via reinforcement learning. IEEE Robot Autom Lett 2022; 7: 3711–3717.

27.

Lepora

Lloyd

. Optimal deep learning for robot touch: training accurate pose models of 3D surfaces and edges. IEEE Robot Autom Mag 2020; 27: 66–77.

Optimizing link-length fitting between an operator and a robot with compensation for habitual movements

Abstract

Keywords

Introduction

Methods

Optimization framework to compensate for habitual movements

Implementation of Digital Annealer

Experimental results

Simulation conditions

Link size compensation simulation

Simulations to compensate for the habitual movements of an operator in 3D space

Experiments to compensate for the habitual movements of operators

Discussion

Conclusions

Footnotes

Acknowledgments

Data availability

Declaration of conflicting interests

Funding

ORCID iD

References