Abstract
Telerobot system plays an important role in executing task under hazard environment. As the computer networks such as the Internet are being used as the communication channel of telerobot systems, varying time delay causes the overall system unstable and reduces the performance of transparency. In this paper, we propose twelve operation modes with different control schemes for telerobot on the Internet with time delay. And an optimal operation mode with control scheme is specified for telerobot with time delay, based on the tradeoff between passivity and transparency properties. We experimentally confirm the validity of the proposed optimal mode and control scheme by using a simple one DOF master–slave manipulator system.
1. Introduction
Telerobot systems are being widely used in various cases in scientific research and peoples' life, such as rescue robot under unknown environment[1–2], rehabilitation robot in the healthcare field[3–4] and a networked telerobot system that allows groups of participants to collaboratively observe live remote environments[5]. To improve the human operator's ability to perform complex task, the interactive force between remote slave manipulator and task environment is often fed back to the local human operator. Currently, more and more computer networks such as the Internet are being used as the communication channel of telerobot systems due to the fact that these with computer networks are easier to implement and have higher flexibility than those with some kind of dedicated private communication channels. However, one problem associated with computer networks is that the communication time delay between the master and the slave is not only significantly large but also keeps changing depending on the network conditions [6]. With force reflection, even a small time delay in the communication can lead to instability due to the non-passivity of the communication line [7].
Several control methods have been proposed to overcome the stability problem associated with varying time delay. Xiong et al. [8] used a predictive display method and man-virtual robot interaction based on augmented reality to control a telerobot. Li et al. [9] built a kind of virtual-environment mode and correct the parameters of the model online. They also used force reflection and object deformation simultaneously to enable the human operator to feel and to observe the influences of interactive force with time delay. Mouhamed et al. [10] presents a robust telerobotic system that consists of a real-time vision-based operator hand tracking system and a slave robot which are interconnected through a LAN. Lu et al. [11] presented a detailed energy-compensating methodology to improve the performance of haptic rendering by the human operator during interactions. Park et al. [12] proposed a modified sliding-mode control algorithm to compensate the varying time delay and to guarantee its nonlinear gain to be independent of the magnitude of time delay. However, those control methods mainly focus on stability robustness of telerobot system with varying time delay, the transparency of the system on the Internet has not further discussed yet.
Although the literatures about control methods for telerobot system with time delay are at least one hundred, only a few discussed the operation modes of telerobot. Sato et al. [13] proposed a concept of intelligent telerobot systems based on geometric world model and pointed out the importance of coordination between the human operator and the slave manipulator and intervention by the operator at various levels, such as servo level, motion level and task level. Yokokohji et al. [14] proposed six operation modes and control schemes for cooperation between manual operation and autonomy function in telerobot system. But they didn't consider the case of time delay.
In this paper, we propose twelve operation modes with different control schemes for telerobot with time delay. And an optimal operation mode is specified for telerobot with time delay. We also give the sequence of mode change to reach the optimal operation mode. We experimentally confirm the validity of the optimal mode and control scheme by using a simple one DOF master–slave manipulator system.
2. System Architecture
The telerobot system can be represented by the block diagram of Fig. 1 and consists of human operator, master and slave manipulators, communication line, and environment. The dynamics of telerobot system on the Internet with time delay is given as follows.

Architecture of telerobot system
where x and f denote position and force; M and B denote mass and viscous coefficient; subscript ‘m’ and ‘s’ denote the master and the slave, respectively; fmd and fsd are the actuator driving forces of master and slave manipulators, respectively.
The Internet as a communication line can be represented by
where T1(t) is the transmission time delay from the master to the slave, and T2(t) is the time delay in the opposite direction. Note that these delays vary with respect to time.
In manual control, which is the most important operation type, human operator commands a velocity forward, through the master manipulator, the communication line and the slave manipulator, to the environment. Likewise, the interactive force between slave manipulator and environment is transmitted back through these blocks to the human operator. Ideally, human operator can feel he is directly touching the remote environment by maneuvering the telerobot system, that is to say, the environment motion ẋe equals to the human operator's motion ẋm, and the human operator's force acted on the master manipulator fm equals to the force applied on slave manipulator by environment fe. Ref. [15] defined the transparency notion as following. A telerobot system is said to be perfectly transparent only when it satisfies the condition
where Zt is the virtual impedance transmitted to the human operator.
When varying time delay arises in the communication line, it not only causes the instability, but also degrades the transparency of the telerobot system. Lawrence [15] pointed out that passivity and transparency are conflicting design goals in bilateral telerobot systems with time delay.
Virtual environment based predictive control can effectively solve the problem caused by time delay of communication. The dynamics of virtual environment model is given as
where, except adding a latter ‘v’ in subscript to denote virtual environment, the dynamics (6)(7) are similar to the dynamics (1)(2). If virtual environment can be established precisely, then the human operator can feel the feedback force fve from the virtual environment without time delay, which is a precise prediction of the delayed force from slave side. In this case, the telerobot system may have optimal transparency and stability.
In autonomy control, slave manipulator can perform some given task such as obstacle avoidance, compliant contact with task etc.
In shared case, the given task is shared among the manual control, predictive control and autonomy control. For example [14], in order to carry a cup filled with water, the orientation is controlled by autonomy function so that the cup orientation keeps horizontal, whereas the cup position is commanded by the human operator.
3. Operation modes
Figure 2 shows a conceptual sketch of a general telerobot system on the Internet. Where A, B, C are switches of operation modes, which connect or disconnect the control loop among human operator, slave manipulator, virtual environment, and slave autonomy function, respectively. D is a switch for model parameter modification in the virtual environment. Those switches could be realized by means of hardware or software.

Switches for operation modes alteration
Let “0” denote the switching off state, “1” denote the switching on state, and “φ” denote the random state of each switch. Thus, there may be exist 64 possible operation modes according to the states of switches A, B, C, D. But, there are twelve operation modes are related to the telerobot on the Internet by eliminating the invalid modes and similar modes of operation.
Free mode (mode F): A1A2B1B2CD=000000
In this mode, the master manipulator could be arranged to a suitable position by the human operator, whereas the slave manipulator is uncontrolled.
Bilateral mode (mode B): A1A2B1B2CD =1φ100φ
Mode B is an essential operation mode in the telerobot system. This mode is used for usual telerobot before adding autonomy function. If there is no time delay in communication, the telerobot system under this mode may be passive and well transparent. If A2 and D are also switched on, then an important function of parameter modification for virtual environment model is added in this mode.
Virtual environment initial mode (mode VI): A1A2B1B2CD=0000φ1
In this mode, the virtual environment is initialized or modified by the model modification function, and the slave force feedback is sent into the model modification function for modifying the parameters of environment dynamics model. This mode is essential for the telerobot system with time delay.
Unilateral mode (mode U): A1A2B1B2CD=1φ000φ
In this mode the master manipulator is used just like a joy-stick without force reflection. This operation model has no problem of instability caused by time delay.
Autonomy mode (mode A): A1A2B1B2CD=00001φ
In this mode, the slave manipulator becomes a full autonomous robot where the human operator cannot intervene the task but can just monitor the task through TV screens. Master manipulator motion become free so that the human operator can move the master manipulator to a convenient posture before intervening the task in other modes.
Virtual environment mode (mode V):A1A2B1B2CD=01010φ
In this mode, the position of the master manipulator maneuvered by human operator is sent to the virtual slave manipulator in local virtual environment, and the force between virtual manipulator and virtual task is fed back to human operator without time delay. If the local virtual environment is completely identical with the remote real environment, then this operation mode could overcome the problem of instability and non-transparency caused by time delay.
Autonomy aided bilateral mode (Mode AB): A1A2B1B2CD =10101φ
In this mode, an autonomy function is added to mode B. The motion command from the autonomy function is mixed into the command from the human operator. Therefore, the human operator can intervene the task performed by the autonomy function and simultaneously he can get the feedback information from the remote environment.
Monitor mode (mode M): A1A2B1B2CD =001010
In this operation mode, slave manipulator is only controlled by autonomy function. But, human operator could feel the feedback force between slave manipulator and task through master manipulator, and monitor the task through TV screens.
Autonomy aided unilateral (mode AU): A1A2B1B2CD =100010
In this operation mode, the human operator can intervene the task without force feedback. The master manipulator is used just like a joy-stick as in mode U. Master manipulator will be free to move or fixed rigidly according to the added autonomy functions. For example, when the autonomy function is force control, the master manipulator will be fixed so that the human operator can command the force through the master manipulator. On the contrary, when the autonomy function is based on position control, the master manipulator will be free so that the operator can modify the position command to the slave arm by moving the master manipulator.
Fusion of force information mode (mode FF): A1A2B1B2CD =111100
In this operation mode, the feedback force from remote slave manipulator and virtual feedback force from local virtual environment are fused before applying on the human operator. This operation mode is usually used in the case of short time delay in communication line. For example, the force applied on the human operator could be a combination of low frequency component of the delayed remote feedback force with high frequency component of virtual force from local virtual environment.
Integrative control mode (mode I): A1A2B1B2CD =110111
In this mode, motion of the master manipulator maneuvered by the human operator is sent to local virtual slave manipulator and remote real slave manipulator as a command simultaneously. The virtual environment produces virtual force, which is a prediction of delayed real force from the slave manipulator, as a feedback force to act on human operator real-time. The remote slave manipulator accepts the delayed position command and track the master manipulator with the aid of autonomy function. The delayed feedback force from remote slave manipulator does not act on human operator, but is used for modifying the model of virtual environment.
Associate Integrative control mode (mode AI): A1A2B1B2CD =110101
we call it associate integrative control mode. The difference between mode I and mode AI is that the autonomy function is not used in the latter.
In the previous work [14], only six operation modes F, B, U, AB, A, AU without time delay were proposed. In this paper, we not only add six operation modes, but also concern the time delay problem.
4. Control schemes
Ref. [14] has discussed the control schemes for six operation modes without time delay. In this section we mainly discuss the added six operation modes and the case of time delay.
Modes B and AB under time delay
Several control schemes have been suggested for bilateral telerobot system when time delay is present in the communication line, such as passive control [6], wave-variable based impedance control [16], and position-error based schemes control [17], etc.
If time delay is constant, we could use the control scheme proposed by Niemeyer [16]. In this control scheme, the communication line equations are defined as
where the characteristic impedance parameter b is strictly positive, which can be chosen arbitrarily. um and us are wave variables.
The controller equations are given as
The above control scheme can guarantee the passivity of bilateral telerobot system with constant time delay. However, this guarantee of passivity comes at a price of reduced stiffness [18]. That is to say this wave-variable based passive control scheme degrades the transparency of telerobot system.
If the communication time delay is varying, we could use the wave-variable based control scheme proposed by Yokokohji [14] to realize mode B. In this control scheme, a compensator is defined to compensate the position drifts due to the time-varying delay as following
where ũs is waveform distorted by varying time delay at the slave side; u′s is an ideal (not distorted) waveform, which is equivalent to um (t – T1); K denotes a positive feedback gain; and tlimit s is defined as
where T̄1 is a standard delay time estimated by any statistical manners. Note that T̄1 means the typical delay time over a certain period of time and do not means the worst case. Therefore, T̄1 may vary according to the condition of the communication line.
Although this control scheme can avoid over degradation of the system performance due to the varying time delay, it does not maintain the passivity rigorously. If we emphasize the passivity of the system, we can use the modified wave-variable based control scheme proposed by Yokokohji [19], but the modified method may degrade the transparency property.
In the Internet-based communication, the time delay is not only varying, but also frequently very large[20]. The slave manipulator must depend on the delayed commands from the master side. Such delayed commands would force the slave manipulator move toward a hard object or an obstacle as before even though at that moment the master side sends commands to stop the slave manipulator or to reduce the speed of the slave manipulator in order to avoid collision. This would result in an intensive impact, which may damage the system. Therefore, the mode AB is necessary for telerobot system under large varying time delay. The control scheme for mode AB could be given by directly adding an autonomy function on the control scheme for mode B. Here, we use a compliance control algorithm proposed by Park [7] as an autonomy function for reducing the impact force.
where Mc, Bc, and Kc are impedance parameters for local compliance; xc is the position when a contact occurs.
Although the addition of autonomy function to mode B could improve the safety and stability of telerobot system under large varying time delay, it causes the displacement between master and slave manipulators.
Mode M
The operation mode M could be easy to realize by choosing an autonomy function proposed by Yokokohji [10] in the slave side and using delayed force feedback control in the master side.
where xd is desired trajectory, and kv, kp are control gains.
Modes VI and V
In modes VI and V, virtual environment provides a predictive visual display and force feedback to the human operator. Because the remote environment and task are often pre-unknown or partly pre-unknown, the key issue in these two modes is how to initialize and modify the parameters of virtual environment model on line by using the delayed feedback information of slave side. Here, we use our previous parameters modification method [21] for the parameters modifying on line as
where λ ∈ (0,1) is modification step length, and e(t) is force error
The initial value of the parameters could be set as
This initial set could guarantee the passivity of telerobot system with time delay when the virtual environment produces predictive feedback force to apply on human operator. Control scheme for master and virtual slave manipulators can be chosen as
where k1, k2, k3 are control gains.
Mode FF
Mode FF is often used in the case of short time delay within 2 seconds. Here, the complementary force feedback control method proposed by Buzan et al. [22] is used, which presents to the master manipulator the sum of a low-pass-filtered delayed force feedback and a high-pass-filtered predictive force feedback.
where H(s) and Hp(s) are low pass filter and high pass filter, respectively.
Mode AI and I
In these modes, Equations (23) and (24) are used for master manipulator control and virtual slave manipulator control, respectively. To overcome the instability caused by varying time delay, Equations (14) and (15) are used for remote slave manipulator control. If the dynamics parameters of remote environment are pre-unknown or partly pre-unknown, then the parameters initialization (22) and modification algorithm (20) are used for initializing and modifying the parameters of dynamics model in virtual environment on line. Adding the autonomy function, such as Equation (17) on the above control scheme for mode AI leads to a control scheme for mode I.
In addition, the control schemes for modes A, F, U and AU under large varying time delay are the same as the control schemes [14] in the case of no time delay.
Note that the sequence of mode change among these twelve modes should be fixed, and the rule for mode change has been discussed in Ref. [14].
5. The optimal operation mode under large varying time delay
Mode V could provide the human operator with predictive force instead of delayed force feedback. So that the virtual environment based predictive control can guarantee the perfect transparency and stability of telerobot system. But this transparency needs precise dynamics model of remote environment. Unfortunately, the telerobot system often work in a pre-unknown or partly pre-unknown environment, we could not establish the precise dynamics model of virtual environment in advance. In that case, the modes B and AB with the modified wave-variable based control schemes [19] and autonomy function [12] are suitable, it could guarantee the passivity and safety when large varying time delay presents in the communication line, but these modes has bad transparency. Considering a reasonable tradeoff between passivity and transparency, the operation mode AI or I is optimal. However, in general we could not let telerobot system work in mode AI or I firstly, so we should use other modes and control schemes at first. The idea of us is to guarantee the passivity firstly under the large varying time delay, and secondly improve the transparency without degrading the stability step by step. Therefore, a procedure of mode change to realize the optimal mode AI or I is given as:
Step 1 Let telerobot system work in mode A, in this mode, slave manipulator adjust its position to a desired position under the autonomy control, in the meantime, the visual and force information of slave manipulator are sent back to master side.
Step 2 Change the operation mode from A to VI. In mode VI, the parameters of virtual environment dynamics mode are initialized.
Step 3 Chang the operation mode from VI to M. In this mode, the human operator arranges the position of master manipulator according to the delayed feedback visual and force information.
Step 4 Let the telerobot system work in mode B (or AB) with the control scheme mentioned above. In this mode, the parameters of environment dynamics model are modified on line by Equation (20).
Step 5 Once the modified parameters satisfy the requirement of precision, change the operation mode from B (or AB) to I. If the virtual environment model is completely equivalent to the remote environment, the operation mode AI could be instead of mode A.
6. Experimental results
We design a one DOF master-slave telerobot system shown in Fig. 3 and Fig. 4 for experiment. The data buffer storage technique is used to simulate the varying time delay here, and the size of buffer is changed according to some random number.

Structure of one DOF telerobot system

Photo of one DOF master-slave telerobot system
Limited by the length of this paper, we only give the experimental result of operation mode AI shown in Fig. 5. Where, the time delay is varying from 11seconds to 12 seconds. The environment is a fixed light foam plastic, the modified parameters of the dynamics model in virtual environment is Kve=1.13Kg/cm, Mve=0, Bve=0.07Kgs/cm.

Experiment result of mode AI
Fig. 5 shows that (1) the telerobot system with varying time delay is robust stable in mode AI; (2) the current motion of the master manipulator is similar to that of the slave manipulator after time delay; and the current force acted on human operator is similar to that of slave manipulator applied by environment after time delay, which implies that in mode AI, the telerobot system with varying time delay is significant transparent.
7. Conclusion
In this paper, we have proposed twelve operation modes for telerobot with time delay, and discuss the control scheme to realize the operation modes associated with time delay. We have also specified the operation mode I or AI is the optimal mode when large varying time delay arises in communication line by using the notions of passivity and transparency. The validity of the proposed optimal mode is demonstrated by experiment of one DOF master-slave telerobot system.
Footnotes
8. Acknowledgments
This research was supported by Natural Science Foundation of Jiangsu Province under grant no.BK2010063 and the Key Project of National Ministry of Education under grant no.708045.
