Sage Journals: Discover world-class research

Abstract

In a robot teleoperation system, humans operate and respond by relying on the content displayed on the video screen. However, unavoidable delays, such as network transmission, cause asynchronous communication between humans and robot and further result in human operation errors such as redundant or duplicate operations or even wrong commands. Users of consumer electronics in daily life are different from users in large-scale industry as they may not be experts or even trained to use the robots leading to new challenges in developing home-used robots. In this study, a module-based remote control system is designed to illustrate the possible human operation errors caused by transmission delay of transmission control protocol/Internet protocol–based network and to provide a user-friendly framework for simulating solutions to improve human errors. This is followed by proposing two target prediction methods, namely the target area predictor and stop position predictor, to decrease the human errors. The experimental results show that target area predictor and stop position predictor effectively improve the performance by reducing task completion time and object trajectory errors by 60% and 36%, respectively.

Keywords

Remote control system transmission delay human computer interaction

Introduction

Robots are used in manufacturing industry and agriculture to reduce labor costs and improve standardization of a work process. Robots replace humans in performing repetitive and dangerous tasks (such as welding, painting, assembly, and testing) or tasks that humans are unable to perform due to size limitation or extreme environments (such as outer space and underwater). Advances in technology and cost reductions in software and hardware development led to the use of robots in daily life. Examples include social robots,¹ healthcare robots,^2,3 and medical robots.^4,5 The technology of autonomous robots is under sustained development. However, all tasks cannot be completed by autonomous robots due to constraints of costs, variability, and flexibility. With the implementation of remote control technology, it is possible to accomplish the tasks that cannot be autonomously performed. To date, several remote control robot applications exist and include remote surgery,^5–7 remote astronautic robot,^8,9 telemedicine, and others.^4,10,11

Previously, most remote control applications in consumer electronics were used to operate devices, such as a television set, DVD player, or other home appliances,¹² from a short distance. Advances in network and mobile devices have led to the increasing use of long-distance remote control applications. However, long-distance remote controls are affected by the transmission speed or network delays, thereby causing inefficient operation or even wrong commands. Consequently, it is essential to solve the delay in the network. There are several existing applications where robots are remotely controlled. However, the operators of these applications include trained experts such as doctors and engineers. It is expected that an improvement in technology will lead to the adoption of the remote control system in specialized fields as well as in daily life. This type of remote control system exhibits many differences from the professional system. In order to make it more accessible to the public, it should be easy to operate such that users need not undergo any tough training processes. In this study, a remote control system that lowers the threshold of operational technique is proposed to ensure that remote control robots are handier for normal individuals.

In the study, the basic structure of a remote control system is first introduced, and the possible human operation errors caused by network delays are analyzed. This is followed by specifying the manner in which a human action predicting method automatically fixes mistakes resulting from human errors. Furthermore, the study describes a virtual environment for the remote control system that can simulate several methods to easily and promptly improve the remote error. The aim of this study involves solving the problem of asynchronous workspace between local and remote environments. Two action predictors are proposed that can predict the conduct of users during a teleoperation in a real network environment. According to the network delay, the real action that is adopted by users is predicted. This is followed by modifying commands sent from users such that the executed result is the same as that expected by users. A human simulator is developed in addition to the prediction model. The simulator simulates the action that a user performs in a remote control system. The simulator aids in measuring the accuracy of the prediction model.

The rest of this article is organized as follows. Section “System overview” gives an overview including an example of transmission delay problem and the system architecture of remote control systems. Section “Proposed solutions” describes our proposed solutions, target area predictor (TAP), and stop position predictor (SPP). To evaluate TAP and SPP, a human simulator is developed and the results are presented in section “Performance evaluations.” The last section is the conclusion.

System overview

Problem description

In a robot teleoperation system, a human operates to respond to the scene observed on a video screen. However, network delays result in a difference between video and actual environment. As shown in Figure 1, the video delay in a video call approximately corresponds to 250 ms with a strong signal, and the video delay approximately corresponds to 3000 ms or more with a weak signal. The video delay causes significant differences between local and remote environments. Thus, operators cannot take correct actions to complete the task. For example, the operator wants to move an object from point $X_{1}$ to point Y. Initially, the operator invokes a command to move the object to a distance of $X_{2} - X_{1}$ . The operator then discovers that object is still at point $X_{1}$ and thus invokes a command to move the object to a distance of $Y - X_{1}$ . The video stream is delayed although the object is actually at point $X_{2}$ . In the end, the object exceeds the expected position as shown in Figure 2.

Figure 1.

Video delay of FaceTime, Skype, and Hangout over WiFi: (a) FaceTime—strong signal, (b) Google+ Hangout—strong signal, (c) Skype—strong signal, (d) FaceTime—weak signal, (e) Google+ Hangout—weak signal, and (f) Skype—weak signal.¹³

Figure 2.

Asynchronous actions between local workspace and remote workspace.

A testing program is designed to record a user’s actions during remote manipulation with network delays. The goal of the mission involves moving the object along a fixed axis to a predefined position. However, the position of the object is inconsistent between local and remote sites due to poor network transmissions as shown in Figure 3. The charts denote the position of the object seen by users, the actions performed by the user, and the real position of the object at the remote site. In the testing program, it is assumed that the video delay corresponds to 1000 ms. At time $t_{0}$ , the user invokes a moving command that causes the object to start moving. However, the user cannot see that the object moves till time $t_{1}$ . The elapsed time between $t_{0}$ and $t_{1}$ corresponds to the video delay. The user wants to move the object to position 400 and finds that the object reaches the goal position at time $t_{3}$ . Thus, the user invokes a stopping command. However, the object actually reaches the goal position at time $t_{2}$ due to a network delay. Thus, the actions performed by the user between time $t_{2}$ and time $t_{3}$ lead to stopping the object stop at an unexpected position. The redundant travel path that the object moves in the given period of time corresponds to operation error.

Figure 3.

Operation errors caused by video delay.

System architecture

In the study, a remote control application is developed and adopted to client/server architecture. Figure 4 shows the system architecture of the application. The operator observed an object scene and moves the object to a specific position by sending transmission control protocol (TCP) commands to the remote application. The remote application parses commands received from the local application and moves the object on the scene. Simultaneously, an object scene capturer and a real-time transport protocol transmitter record the scene and periodically send the object scene to the local application.

Figure 4.

System architecture of remote control applications.

In order to solve the asynchronous workspace between local and remote, two methods were proposed to improve human control error. The first method is termed as a TAP. The TAP is used to estimate the bound of the target area. After obtaining the bound, the object is limited in a small area. The second method is termed as an SPP. The stop position is predicted, and this corresponds to the operator’s goal. The object velocity slows down while moving the object close to the point. Hence, the operator has more time to react to the video delay.

It is assumed that the network delay time in the system is known as shown in Figure 5. Thus, $D e l a y_{v}$ denotes the delay of streaming video transmission from remote site to local site, $Dela y_{c}$ denotes the delay of commands transmission from local to remote site, constant K denotes human reaction time, and $Dela y_{execution}$ denotes time required to achieve the command at the remote site. In the study, it is assumed that the $Dela y_{c}$ , K, and $Dela y_{execution}$ correspond to zero.

Figure 5.

Analysis of delays.

Proposed solutions

Target Area Predictor (TAP)

In the remote control assumption, it is not necessary for the operators to perform mutual actions to complete a mission. For example, there is a task that involves horizontally moving the object toward its right from position $X_{1}$ to position $X_{2}$ . The only necessary action in this task involves moving toward the right. However, the operators cannot accurately move the object to position $X_{2}$ due to a network delay. The movement of the object may exceed the target position. Thus, the operators have to move the object toward the left. However, moving the object toward the left is a redundant action in this task. The purpose of the TAP involves reducing the redundant actions.

As mentioned above, the only necessary action in the task involves moving toward the right. It is surmised that the object exceeds the target position when the operators move the object toward the left. As shown in Figure 6, the object is at position $X_{1}$ in the beginning. At time $t_{1}$ , the operator moves the object toward the right from position $X_{1}$ to position $X_{2}$ . At time $t_{2}$ , the operator moves the objects toward the left from position $X_{2}$ to position $X_{3}$ . Hence, it is inferred that target position that the operator wants to reach does not exceed position $X_{2}$ . At time $t_{3}$ , the operator moves the object toward the right from position $X_{3}$ to position $X_{4}$ . Consequently, it is expected that the target position that the operator wants to reach is between position $X_{3}$ and position $X_{2}$ .

Figure 6.

An example of setting predicted target area.

When the expected target position interval is realized, the object movement is limited in the range between the intervals. As a result, the object trajectory error reduces.

Recovery mechanism of TAP

There is a worst case that the operator cannot move the object to correct position with TAP constantly when the predicted target area is incorrect. As shown in Figure 7, the object is at position within incorrect predicted target area. However, the correct target position that the operator wants to reach corresponds to position Y. The object cannot be moved to position Y because of TAP boundary. Hence, a recovery mechanism that can enlarge the expected interval is designed.

Figure 7.

Incorrect prediction target area.

With TAP, the object stops when it reaches the limited bound in the remote site. Nevertheless, the operator at that moment observes that the object has not reached the bound. Thus, the operator continuously moves the object. However, it is inferred that the operator wants to move the object to exceed the bound if the operator is still moving the object after a period of time when the object reaches the bound. The period of time is calculated from the network delay. As shown in Figure 8, the object is at position $X_{1}$ in the beginning, and the operator begins to move the object toward the right. At time $t_{1}$ , the object in the remote site reaches the bound. Simultaneously, the operator observes that the object is still at position $X_{1}$ . If the operator still moves the object after time $t_{2} + Dela y_{v}$ , then it is inferred that the operator wants to move the object such that it actually exceeds the bound because the operator observed that the object has reached the bound at time $t_{2} + Dela y_{v}$ . Hence, the operator still moves at this moment, and it is inferred that the operator wants to move the object such that it exceeds the bound. Finally, the bound that was set earlier is canceled. In the results, this solves the problem of setting the wrong interval. The functions of TAP are described in Algorithm 1.

Figure 8.

An example of TAP recovery.

Algorithm 1. Target area predictor (TAP).
1: function UpdateTABoundary()
2: if CurrentActionDirection is opposed to LastActionDirection then
3: Boundary of LastActionDirection ← LastStopPosition
4:
5: function TABoundaryCheck()
6: if Object is at boundary of
CurrentActionDirection then
7: return out of TA boundary
8:
9: function RecoverTAPError()
10: CalculatedTime ← $CurrentTime - Dela y_{v} - Dela y_{c}$
11: if Object has already reached bound at time CalculatedTime then
12: Remove boundary of CurrentActionDirection

Stop Position Predictor (SPP)

In addition to the TAP, a method to predict the target position that the operator wants to reach is also proposed. With this predictor, the control system can adjust the velocity of the object such that it slows down in a particular range. Subsequently, the operator moves the object accurately as shown in Figure 9.

Figure 9.

Reduce object’s velocity in SPP.

It is assumed that the operator does not realize that there are network delays, and thus the operator stops moving the object when the object is at the target position. However, the object does not stop at the target position due to a network delay. The object stops at the target position plus a redundant displacement. The redundant displacement $S_{redundant}$ is defined as follows

S_{redundant} = Dela y_{v} \times V_{o}

(1)

where $Dela y_{v}$ denotes streaming video delay and $V_{o}$ denotes the object’s velocity because other delays are assumed to be zero in this study. Therefore, we can predict the target position $P_{target}$ as follows

P_{target} = P_{static} - S_{redundant}

(2)

where $P_{static}$ denotes the actual stop position of the object.

Figure 10 is considered as an example. The operator wants to move the object to target position TP such that he or she starts moving the object toward the right in the beginning. At time $t_{2} + Dela y_{v}$ , the operator finds that the object is at TP and therefore stops moving the object. However, a network delay exists such that the object actually stops at time $t_{4}$ , and the object finally reaches position LSP. The distance between TP and LSP corresponds to redundant displacement as shown in Figure 11. Equation (2) is used to obtain the predicted target position as follows

TP = LSP - S_{redundant}

(3)

Figure 10.

An example of SPP.

Figure 11.

SPP with compensation.

SPP with regulation

We found that an increase in the number of operations causes the operators to move more accurately. In other words, the operators adapt the network delay gradually in the process of task and therefore stop moving the object before the object reaches the target position as shown in Figure 11. Thus, this situation should be included in the previous equations.

We developed a test program in which human subjects are required to complete a task that involves moving an object to a particular position. The human subjects only have a chance to move the object in every iteration. Human subjects are required to repeat the task 10 times. The human subjects include eight computer science students. In the first experiments, there is no network delay and the deviation from the target position to the final position was calculated. As shown in Figure 12, the average of deviation without network delay is almost fixed that implies a fixed operation error existing when human operates machine even though there is no network delay.

Figure 12.

Average of operation errors with/without network delay.

In the second experiment, the network delay is considered and the average of deviation with network delay decreases because the operators adapt to the effect of network delay and thus move the object more accurately. Based on this experiment, a regulation function $C (n)$ that represents the manner in which the operators are familiar with network delay is defined as follows

C (n) = α n^{β}

(4)

where n denotes the number of operations, and $α$ and $β$ denote the coefficient equals to 0.5982 and constant equals to −0.769, respectively.

The static position of the object with operator regulation $P'_{static}$ is defined as

P'_{s t a t i c} = P_{s t o p} + S_{r e d u n d a n t}

(5)

and

Compensation = [1 - C (n)] * S_{redundant}

(6)

where $P_{stop}$ is the screen position of object when operator sends stop command.

So the modified predicted target position $P'_{target}$ is defined as follows

\begin{matrix} {P'}_{target} = P_{stop} + Compensation \\ = P_{stop} + [1 - C (n)] * S_{redundant} \\ = P_{stop} + S_{redundant} - C (n) * S_{redundant} \\ = {P'}_{static} - C (n) * S_{redundant} \end{matrix}

(7)

The functions of SPP are described in Algorithm 2.

Algorithm 2. Stop position predictor (SPP).
1: function PredictStopPosition()
2: RedundantDisplacement ← $Dela y_{v}$ * $V_{o}$
3: C ← $α$ * $NumberOfOperatio n^{β}$
4: if CurrentActionDirection is opposed to LastActionDirection then
5: PredictedPosition ← LastStopPosition - C*RedundantDisplacement
6:
7: function Update Velocity()
8: if Object is near by PredictedPosition with a particular range then
9: Slow down the object

Performance evaluations

Human simulator

In order to measure the performance of the proposed solutions, a human simulator was developed. In the simulator, an action model of the tasks performed by operators in a remote control task was designed. The results in Figure 12 indicated that the operators cannot move the object to a specific position accurately in a smooth control. Thus, the operators are unable to precisely move the object even in an environment without network delay. It is assumed that the operator wants to move the object to a target position $P_{target}$ , and thus there exist a few operation errors $E_{operation}$ such that the object is not precisely at the target position. The simulator uses a random number generator that follows a normal distribution, $ψ ()$ , to generate a position to which the object moves to, and the screen position of operator sending stop command $P_{stop}$ is defined as follows

P_{stop} = ψ (P_{target}, E_{operation})

(8)

After obtaining $P_{stop}$ from the generator, it should be summed with a redundant displacement caused by network delay. As noted earlier, the object exceeds the target position due to a network delay. The redundant displacement $S_{redundant}$ is defined as follows

S_{redundant} = Dela y_{v} \times V_{o}

(9)

where $Dela y_{v}$ denotes the streaming video delay and $V_{o}$ is the object’s velocity. The actual object position after this action occurs at real position $P_{real}$ , which is defined as follows

P_{static} = P_{stop} + S_{redundant}

(10)

However, the results show that the operators automatically fix the target position $P_{target}$ such that the target position $P_{target}$ is equal to the static position $P_{static}$ . Thus, a compensation function $C (n)$ is designed that represents the manner in which the operators are familiar with the network delay. An increase in the number of operations increases the accuracy accordingly.

The original mean of the normal distribution in equation (7) corresponds to the target position $P_{target}$ . The mean is subtracted by the compensation function $C (n)$ . Thus, the modified stop position $P_{stop}^{'}$ is defined as follows

P_{stop}^{'} = ψ (P_{target} - C (n), E_{operation})

(11)

The modified object static position $P_{static}$ is defined as follows

P_{static} = P_{stop}^{'} + S_{redundant}

(12)

The human simulator aids in measuring the performance of the proposed solutions in the remote control system.

Simulations

The task of our simulations is to move an object from a starting position to a specific target position without restriction on number of operations. The human simulator is used to execute the task for 100 times. Algorithm 3 shows the flow of controlling the movement of an object by adopting TAP or SPP or both. First, there is a boundary checking function to ensure that the object will stay in the movable area for each operators command. Then, two flags, enableTAP and enableSPP are used to decide whether TAP or SPP or both are applied to decide the movement of object. The functions TAP and SPP are described in Algorithm 1 and 2, respectively.

Algorithm 3.
1: procedure UpdateObjectPosition()
2: if !BoundaryCheck() is out of boundary then
3: return
4: if enableTAP then
5: UpdateTABoundary()
6: RecoverTAPError()
7: if TABoundaryCheck() is out of boundary then
8: return
9: if enableSPP then
10: PredictStopPosition()
11: UpdateVelocity()
12: MoveObject()

Algorithm 4.
1: function BoundaryCheck()
2: if Object is at boundary of CurrentActionDirection then
3: return out of boundary
4: return success

Results and analysis

We consider two performance metrics: task completion time and redundant object trajectory. Task completion time is the length of time starting from the operators first command until the object is placed in the target position correctly. Redundant object trajectory is the difference between the total trajectory length and the shortest path from starting position to target position. The total trajectory is defined as the accumulated movement length of the object starting from the first operators command until the object is placed in the target position. Without loss of generality, the redundant object trajectory is presented by pixel/V₀.

Figures 13 and 14 show the performances on task completion time and trajectory errors, respectively. Enable TAP or SPP or both are compared with original which has no improved methods adopted. From Figures 13 and 14, we can see that TAP cannot shorten the task completion time because TAP only limits the objects movement boundary. The benefit of TAP is that the trajectory error of the objects motion trail is reduced by 8%. However, SPP improves both task completion time and trajectory errors. This is because SPP decreases the velocity of object to reduce the impact of network delay so that the operator can control the object more precisely. The results indicated that the task completion time is reduced by 36% and the trajectory error is reduced by 60% with SPP.

Figure 13.

Performances on task completion time.

Figure 14.

Performances on object trajectory errors.

Conclusion

A remote control system is developed in this study. The system can be used to observe the actions adopted by operators during remote manipulation. A network simulator is also adopted in the system to ensure that observations are more realistic. In the system, a task is designed that requires the operator to move an object to a specific position. Data related to objects, such as motion trail, were collected in the process of the task, and the completion time of a task is measured.

A human simulator was also developed in addition to the remote control system. The simulator generates control commands that can achieve the above-mentioned task. A human action model proposed in the study is used in the simulator. Two experiments were designed to observe the performance of the operators in remote manipulation. The simulator allows examining the correctness of the proposed solutions in the remote control system. Furthermore, the performance of the prediction model was also evaluated.

Future studies will involve collecting more realistic human data. In the study, only human data from a specific group, namely computer science students, was collected. Subjects in other groups should be invited. Thus, the human simulator can act in a manner similar to a real human. The advantage of improving the human simulator involves making the proposed solutions more convincing. To date, the proposed solution was only adopted in a virtual environment. The prediction model should only be used in a real environment such as robot arms.

Footnotes

Handling Editor: Stephen D Prior

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

References

Dario

Guglielmelli

Allotta

Robotics in medicine. In: Proceedings of the IEEE/RSJ/GI international conference on intelligent robots and systems: Advanced robotic systems and the real world, Munich, 12–16 September 1994, pp.739–752. New York: IEEE.

Charles

Ronald

CM.

Robotics. Telemed E-Health 2015; 21: 695–696.

Volosyak

Kouzmitcheva

Danijela

et al . Improvement of visual perceptual capabilities by feedback structures for robotic system FRIEND. IEEE T Syst Man Cy C 2005; 35: 66–74.

Elhajj

Fung

et al . Supermedia-enhanced internet-based telerobotics. P IEEE 2003; 91: 396–421.

Haidegger

Sándor

Benyó

Surgery in space: the future of robotic telesurgery. Surg Endosc 2011; 25: 681–690.

DiMaio

Hanuschik

Kreaden

The da Vinci surgical system. In: Rosen

Hannaford

Satava

(eds) Surgical robotics. Berlin: Springer, 2011, pp.199–217.

Marescaux

Leroy

Gagner

et al . Transatlantic robot-assisted telesurgery. Nature 2001; 413: 379–380.

Hirzinger

Brunner

Dietrich

et al . ROTEX-the first remotely controlled robot in space. In: Proceedings of the IEEE international conference on robotics and automation, San Diego, CA, 8–13 May 1994, pp.2604–2611. New York: IEEE.

Yoon

Goshozono

Kawabe

et al . Model-based space robot teleoperation of ETS-VII manipulator. IEEE T Robotics Autom 2004; 20: 602–612.

10.

Lam

Boschloo

Mulder

et al . Artificial force field for haptic feedback in UAV teleoperation. IEEE T Syst Man Cy A 2009; 39: 1316–1330.

11.

Sanders

Comparing ability to complete simple tele-operated rescue or maintenance mobile-robot tasks with and without a sensor system. Sensor Rev 2010; 30: 40–50.

12.

Topping

An overview of the development of Handy 1, a rehabilitation robot to assist the severely disabled. J Intell Robot Syst 2002; 34: 253–263.

13.

Liu

et al . “Can you SEE me now?” A measurement study of mobile video calls. In: Proceedings of the IEEE conference on computer communications (INFOCOM), Toronto, ON, Canada, 27 April–2 May 2014. IEEE. DOI: 10.1109/INFOCOM.2014.6848080.