A novel shared control algorithm for industrial robots

Abstract

Human management of robots in many specific industrial activities has long been imperative, due to the elevated levels of complexity involved, which can only be overcome through long and wasteful preprogrammed activities. The shared control approach is one of the most emergent procedures that can compensate and optimally couple human smartness with the high precision and productivity characteristic to mechatronic systems. To explore and to exploit this approach in the industrial field, an innovative shared control algorithm was elaborated, designed and validated in a specific case study.

Introduction

Collaboration between robots and humans is one of the major topics faced by scientists in recent years: the main issue being to obtain specific goals aimed at improving the sustainability of many processes, especially in industrial contexts. The complementary relationship between human and robot capabilities has already been demonstrated: the smart human approach can be successfully coupled with robots’ ability to deal with large amounts of data arising from the work context. ¹ In this view, from a personal computer (PC) to autonomous companion robots, the evolution of mechatronic systems has revealed the incredible possibilities that such interaction can bring in many fields. It has been shown how a shared initiative can facilitate the achievement of specific goals for indoor search and rescue or exploration tasks, in particular in hazardous environments. ^{1 –4} With reference to the latter applications, human–robot interaction (HRI) was instrumental in rescue operations following the 9/11 terrorist attack. ⁵ However, one shortcoming of this approach is the limited number of ‘slave sides’ that a single human can manage at the same time. ⁶ Hence, the development of more efficient HRI management systems is based on autonomy levels, ⁷ the most widely used of which is known as teleoperation: a ‘slave robot’ is a passive instrument to which high-level decisions are delegated by its human operator. Mechatronic systems are, consequently, only means for optimizing particular features, such as reducing tremors or improving precision. ^{8 –16} This, for example, is the case in innovative devices used for minimally invasive surgery or rehabilitation ¹⁷ where surgeons or users replicate their movements in the working area. These tools can be, for example, haptic interfaces with force feedback control ^{18 –24} or simply joysticks and/or space balls. ^25,26 In addition to this, augmented reality can enhance the sensation of involvement in the action even though the working area is distant from the master control side. This strategy definitively has many advantages in terms of safety, but also entails complications in terms of complexity of the whole system. ²⁷ A very interesting innovative approach is represented by a novel interpretation of shared control (SC), which delegates some of high-level decisions to the robot side, such as mapping static environments ^{28 –30} or handling dynamic ones ³¹ in order to relieve humans of some aspects. This kind of collaborative control of robots implies an increasing level of system autonomy that can take over leadership, to the detriment of humans, for accomplishing specific tasks. ¹ For example, an interesting application optimized the grasping and handling of specific objects in a structured environment: the shared approach allowed the compensation of the limited human precision using the robot accuracy and precision combining smart capabilities of the operator using specific HRI. ³² Nowadays, industrial robotics applications too are moving towards SC approaches. ³³ The industrial robot is set to become a valuable assistant for workers rather than being ‘simply’ an intelligent system used in special cells or in dangerous areas designed to substitute humans in specific contexts. Especially in flexible industrial automation, humans and robots can be complementary, and a team of this type can provide exceptional performance in terms of accuracy, reliability and flexibility. Moreover, both inexperienced and professional operators are expected to benefit from an SC approach. ³⁴

The aim of the research shown in this article was to design a new algorithm for industrial robotics processes using the SC approach. In particular, the algorithm can be used as a model to design any number of industrial processes involving robots. The innovative shared control algorithm for industrial robotics processes (SCAIRPs) shown in this article become a new paradigm for the development of anthropomorphic industrial robotic processes, using the advantages of automation with ‘active supervision’ on the part of users. With this strategy, a mechatronic system can execute its task with the human operator interacting and modifying the robot’s preprogrammed tasks while the process is actually running.

The following sections present the SC algorithm, its design, the HRI used in the algorithm and some possible applications.

SC algorithm

In industrial processes, robots are mainly used for planned scenarios in which preprogrammed tasks are commonly used. The complexity of programming is only one of the major hurdles preventing automation using industrial robots, especially for non-repeatable processes (e.g. welding or painting unknown objects, etc.). ³³ Preprogrammed industrial robots do not have, indeed, enough flexibility to adapt their behaviour in a dynamic environment. Furthermore, human operators cannot transfer their manual ability or experience to robots. These issues can be overcome by a man–machine interactive SC approach: the robot can provide the main task (e.g. desired motion), while the operator changes the robot’s main task guaranteeing adaptive feedback and experience. ^9,34

Taking into account the features owned by industrial robots, this article describes the SCAIRPs as a general purpose design for SC applications. SCAIRP is mainly based on the following considerations:

SCAIRP is designed for real-time applications, namely applications that the user senses as current, without latency.

SCAIRP requires an HRI capable of managing and controlling specific robot’s degrees of freedom (DOFs).

SCAIRP can be applied to any industrial robot in which the controller can provide at least one communication channel for external applications.

External path planning

SCAIRP can be defined as an external path planning, running on an external PC, which communicates with the robot controller giving a position command (PCMD) without directly managing robot dynamic constrains (Figure 1).

Figure 1.

Block diagram showing the application of SCAIRP in planning a specific path. SCAIRP: shared control algorithm for industrial robotics process.

The purpose of SCAIRP is to plan a tool robot path combining, in real time, a preprogrammed robot path with online operator inputs (i.e. motion variations). The algorithm output (i.e. planned tool robot path) consists of a sequence of Cartesian points in the robot operating space. These points are the input for the robot controller that is devoted to generate the final robot trajectory. In the following SCAIRP description, the term ‘path’ is used instead of ‘trajectory’ to underline as this approach is focused on the geometric description of the robot tool movements, using fixed speed v and Δt in the motion equations. In this prospective, planning a path means combining its geometric description with a uniform motion equation: interpolating a discrete set of points imposing environment and dynamic constraints.

The desired path can be described as a parametric curve γ in R³

$\gamma =\left\{\left(x\left(t\right),y\left(t\right),z\left(t\right)\right)\right\},t\in \left[t_1,t_2\right]$

1

As shown in Figure 2, O₁ is the origin of the curve and s is the curvilinear abscissa that links O₁ with a generic point P(t) with coordinates $x\left(s\right),y\left(s\right),z\left(s\right)$

$s=s\left(t\right)=\pm \underset{-t_0}{\overset{t}{{\displaystyle \int }}}\sqrt{\left({\left[\frac{dx(t\prime )}{dt}\right]}^2+{\left[\frac{dy(t\prime )}{dt}\right]}^2+{\left[\frac{dz(t\prime )}{dt}\right]}^2\right)}dt\prime$

2

$P\left(t_0\right)=O_1$

3

Figure 2.

Generic parametric curve γ in R ³.

With the inverse function $t=t\left(s\right)$ of $t\to s\left(t\right)$ , it is possible to define a normal parametric representation of the curve with tangent versor $\widehat{\tau }\left(s\right)$ applied in P(s) with $x\left(s\right),y\left(s\right),z\left(s\right)$ coordinates

$\gamma =\left\{\left(x\left(s\right)=x\left[t\left(s\right)\right],y\left(s\right)=y\left[t\left(s\right)\right],z\left(s\right)=z\left[t\left(s\right)\right]\right),s\in \left[s_1=s\left(t_1\right),s_2=s\left(t_2\right)\right]\right\}$

4

$\widehat{\tau }\left(s\right)=\frac{\frac{dP\left(s\right)}{ds}}{\left|\frac{dP\left(s\right)}{ds}\right|}$

5

By defining $s=s\left(t\right)$ as the function that describes the position of P at time t on the curve, the motion of a generic point can be described by knowing the curve and s(t). Indeed, by knowing P = P(s) and $s=s\left(t\right),$ the parametric representation of P can be found as follows

$P=P\left(t\right)=P\left[s\left(t\right)\right]$

6

The velocity vector of P, at time t, coincides with the derivative of the displacement vector $O_{1}P\left(t\right)=OP\left(t\right)-O{O}_1$

$v\left(t\right)=\frac{d{O}_{1}P}{dt}=\frac{dOP}{dt}=\dot{x}\left(t\right)\widehat{i}+\dot{y}\left(t\right)\widehat{j}+\dot{z}\left(t\right)\widehat{k}=\dot{P}\left(t\right)=\frac{dP\left[s\left(t\right)\right]}{dt}=\frac{dP\left(s\right)}{ds}\frac{ds\left(t\right)}{dt}=\dot{s}\left(t\right)\left|\frac{dP\left(s\right)}{ds}\right|\widehat{\tau }$

7

$\dot{s}\left(t\right)=\pm \sqrt{{\dot{x}}^2\left(t\right)+{\dot{y}}^2\left(t\right)+{\dot{z}}^2\left(t\right)}$

8

$s\left(t\right)=\underset{t_0}{\overset{t_1}{{\displaystyle \int }}}\dot{s}\left(w\right)dw+s_0$

9

In a uniform motion, having the position of P at time t and v(t), the motion of the point can be determined by the following equation

$P\left(t\right)=P\left(t_0\right)+\underset{t_0}{\overset{t}{{\displaystyle \int }}}v(t\prime )dt\prime$

10

Furthermore, setting L as the s(t) length, v as the constant speed in a uniform motion and $\Delta t= \frac{t_1-t_0}{\text{NOC}}$ as the time cycle, the number of cycle (NOC) required to execute the path can be calculated as

$\text{NOC}= \frac{L}{v\Delta t}$

11

The path segments Δs such that $L={\displaystyle \sum }_{i=1}^{\text{NOC}}\Delta s_i$ can be calculated as

$\Delta s= \frac{L}{\text{NOC}}$

12

Algorithm design

First, SCAIRP design requires a robot controller able to receive motion movement commands in a fixed period Δt. SCAIRP can calculate and send a new command to robot controller every Δt period. The command is classified as PCMD referred to robot end-effector position in the operating space (robot Cartesian space). The controller will execute in the fixed time Δt each PCMD moving its tool from current position to the new position (Equation (10)).

SCAIRP can manage the main task of the robot and human operator’s corrections/variations. In this way, thanks to external PC computational resources, SCAIRP can achieve the real-time interaction between the robot and the human operator.

Figure 3 shows the algorithm flow chart.

Figure 3.

SCAIRP flow chart. SCAIRP: shared control algorithm for industrial robotics process.

The algorithm is composed of two cycles as follows:

Main cycle (MC): In this cycle, the robot’s main task (motion path) is calculated and conveniently managed to be processed into the ‘subcycle NOC (SCN)’. In this phase, NOC and Δs are defined (Equations (11) and (12)). This cycle can be activated or ended thanks to two possible transactions in MC block. In order to manage the cycle, the Boolean variable loop can be used as user input.

‘MC’ to SCN: This transaction activates the calculation of a new robot path, selected by the operator (loop = true). The first step for the new path calculation is related to the definition of the robot path features, as described in the section ‘External path planning’. In this phase, the algorithm initialization and the definition of system constraints are performed. The former consists to set properly all variables used into the algorithm and the latter defines environmental and dynamic constraints for the path planning.

MC to ‘End’: This transaction is activated by the operator to end the process (loop = true).

• SCN: The PCMD is created in this cycle. In detail, it combines precalculated Δs with the human input calculating the PCMD, evaluates new positions, sends the PCMD to the controller and recalculates the new Δs. The NOC iterations can be modified within the SCN in function of human inputs. As described earlier for the MC, SCN can be activated or ended by two possible transactions using a loop counter.

Human–robot interaction

The SCAIRP approach detailed in this article is designed for real-time and online applications. It can bring relevant advantages if used in combination with HRIs such as haptic interfaces. Hannaford et al. ³⁵ reported drastic reduction time for the accomplishment of a peg-in-hole task; using the feedback provided by the haptic interface, the operator precisely controlled the end-effector position, while the robot controlled the positioning of the arm in the space. The SCAIRP haptic HRI can be used in different manners as follows:

Workspace mode: The use of an interface with the same number of robot axes allows to map the robot workspace in the HRI workspace. Consequently, the human operator can fully manage the robot in a manual mode.

Hybrid mode: By taking into account the robot’s main task in an SC approach (e.g. automatic motion along a line path), the HRI workspace can be used to move the robot in a different way and/or to change task parameters (e.g. end-effector velocity). It is therefore possible to move the robot along a relatively long path without losing resolution in HRI movements (i.e. robot motion variations).

Experiments

The algorithm presented in the previous section was implemented in a complex mechatronic system and tested in a welding laboratory held by GE Oil & Gas in Massa (Italy). A specific embodiment was identified and designed: as for the robot unit, we chose a Mitsubishi Industrial Robot MELFA RV-13FM-D (six-axis) with Mitsubishi Industrial Controller CR 750-D due to his technical specifications (see ‘SC algorithm’ section – SCAIRP requirements). Concerning the human operator side, a haptic HRI, the Phantom Omni, commercialized by Geomagic® (Rock Hill, SC 29730 USA), was selected on the basis of its features, especially its high flexibility and the force feedback option. A HP Z600 Workstation, equipped with dedicated hardware (Quad-Core Processor Intel Xeon E5620 at 2.4 GHz, 12 GB of RAM, Samsung Solid State Drive 250 GB EVO 850) to achieve real-time applications described as follows, was used as an external PC. Figure 4 shows the whole system used for the tests. ³⁶

Figure 4.

SCAIRP testing system. SCAIRP: shared control algorithm for industrial robotics process.

Motion speed variation

In order to test the algorithm initially, a simple experiment was designed (T1). The aim of T1 was to implement the SCAIRP for one robot DOF (motion speed) using MATLAB® simulations. The robot trajectory was defined as a single linear path on x-y plane with a fixed speed along x-axis. The human operator, however, was able to change the speed using the Phantom haptic HRI using a hybrid mode approach (see ‘Human–robot interaction’ section). T1 was conducted using the software-in-the-loop approach with the following simulation setup:

constant $v=500\frac{mm}{\text{min}}$ ;

cycle time $\Delta t= 7 ms$ ;

linear path A start point $\left(0,0\right)$ , end point (50,0);

linear path B start point (0,0), end point (5,0);

simulation of haptic HRI inputs using a random function.

Figure 5 shows Δs function of NOC with L = 50 mm (a) and L = 5 mm (b). In both graphs, the red and blue lines describe Δs increases with and without speed variations. The simulation shows that the SCAIRP can plan a trajectory dividing the path in many increases of Δs relating to the motion speed: the SCN, in the designed algorithm, recalculates the new Δs evaluating human inputs.

Figure 5.

SCAIRP test simulation changing motion speed. SCAIRP: shared control algorithm for industrial robotics process.

T1 simulation permitted progression to an experimental test using a completed system in an industrial process.

Changes of trajectory

These tests (T2) were designed in order to evaluate the robustness of the real-time algorithm in an industrial process. A gas tungsten arc welding (GTAW) process was selected on the basis of its reduced number of parameters requiring control. The welding apparatus is composed of a Mitsubishi six-axis anthropomorphic robot with a GTAW torch positioned on the end effector and a Phantom Omni as the haptic HRI used by a human operator.

SCAIRP was implemented mainly with the following software:

Main external robot controller (MERC): This software was performed using Dev C++ (TDM-GCC Compiler 4.9.2), and it is based on the Mitsubishi MXT real-time protocol. It is used to control and to monitor the robot position and dynamics with the Transmission Control Protocol / Internet Protocol (TCP/IP) standard protocol. MERC completely manages SCAIRP model: planning a tool robot path combining, in real time, a pre-programmed robot path with online operator inputs; communicating with the robot controller giving a calculated PCMD.

MELFA demo: This software manages the Mitsubishi controller. It was written in MELFA-BASIC V language, and it operates as a bridge between the robot and MERC software.

Phantom demo (PD). This software is dedicated to the haptic HRI management. It was written in C++ using the OpenHaptics toolkit supplied by Geomagic.

The PD software handles HRI commands with a cycle time lower than 7.1 ms by taking into account the MXT real-time protocol cycle time (i.e. 7.1 ms), in order to assure the required interaction between robot motion and human input. For the previous reason, T2 was performed with a haptic HRI clock frequency of 1 kHz. Figure 6 shows the two main clock signals of T2: Mitsubishi MXT real-time protocol clock signal and haptic HRI clock signal.

Figure 6.

Test 2 clock cycles.

The robot’s main task was defined as a single linear path, whereas the human operator controls were devoted to changing speed, height of welding (defined as the distance between the welding plan and the torch) and the possibility of performing the ‘welding waving movements’: orthogonal motion with respect to the linear robot trajectory. These pattern fluctuations shown in Figure 7, formally called weaving action, are representative to the strategy adopted by the welder to produce a good welding joint, especially between thick plates. This approach allows, indeed, a strong tie between the basic parts, improving at the same time mechanical properties, ³⁷ as shown in Figure 7. This latter shows the output data of MERC. The human operator is able to cooperate with the robot using the Phantom haptic HRI with the hybrid mode approach (see ‘Human–robot interaction’ section). This test was carried out without any variation of the speed along the x-axis coupled with a weaving action, performed by the welder, at constant speed along the y-axis. The plot shows a latency between the actual trajectory output of the Mitsubishi robot and the actual human haptic input. This behaviour is linked mainly with the decision to improve HRI usability by setting a constant speed along y-axis. Thanks to this solution, the welder can manage the weaving with variable speed and offset: the former is the speed along the x-axis, while the latter is the orthogonal offset with respect to the linear robot trajectory given by the haptic HRI.

Figure 7.

Test 2 path monitoring.

As described earlier, SCAIRP can be used in two different manners: hybrid mode and workspace mode (i.e. the most widely used approach is the teleoperation: a slave robot is a passive instrument to which high-level decisions are delegated by its human operator). The comparison between these operating modes highlights two main advantages in using the hybrid mode which approach makes possible to

delegate to the robot a movement along one axis: it allows welding workpieces with larger dimensions than the HRI workspace;

change the HRI workspace scaling without linking the weaving movements resolution with the welding speed along the welding joint direction. This approach allows to maintain a constant speed throughout the welding process, facilitating the welder to produce a good welding weaving. This advantage is shown in Figure 8: Figure 8(a) shows the result of a welding following a workspace mode, while Figure 8(b) depicts the outcome of the hybrid mode approach. In this latter, it is possible to highlight the first of the track made with a constant speed guaranteed by the robot control (Figure 8(a′)) and, in Figure 8(b′), the outcome arising from the combination of the robotic speed control with the controlled weaving movement given by the human operator.

Figure 8.

Welding weaving movements using the (a) workspace and (b) the hybrid modes. (a′) Welding joint made with the constant speed movement performed by robot. (b′) Welding joint with the constant speed movement performed by robot combined with the weaving movements given by the human operator.

Conclusions

The approach described earlier can bring significant advantages in industrial environments. As reported in the example of GTAW welding process, it can help the operator in performing specific tasks achievable, to date, only by manual or preprogrammed approaches. Consequently, SCAIRP could be advantageous in terms of work efficiency and process sustainability with the operator managing only specific high-level tasks leaving others (e.g. basic trajectory tracking and tremor suppression) to the robot. It is important to stress that this improvement of artificial intelligence neither overwhelms nor substitutes worker activities and supervision: humans are always in the centre of the loop. On the contrary, this approach relieves operators of tiring duties, which is a key element in improving work safety and sustainability.