Development and experiment of the Internet-based telesurgery with MicroHand robot

Abstract

Robot-assisted telesurgery can perform complex surgical manipulations from remote locations. Against the requirements of remote controlling for minimally invasive laparoscopic surgery robot, a prototype telesurgery system integrated on the existing “MicroHand S” robot was built. The prototype worked with the Internet, and a telesurgery communication protocol was set up based on Transmission Control Protocol/Internet Protocol. The stereo images of laparoscopy were transmitted by a hardware-based H.264 encoder/decoder. A demonstration environment of robot-assisted remote minimally invasive surgery between the medical robot laboratory of Tianjin University and the visual surgery laboratory of the PLA Rocket Force General Hospital, which is about 150 km away, was set up. Control signal and laparoscope image transmission test between these two places were conducted. After the pegboard task and knot-tying task of phantom training, the first remote gall bladder removal surgery on the sow of China was successfully performed, on the basis of remote surgery robot system integration. The experimental results show that the developed prototype telesurgery system has enough remote control performance and operability for telesurgery.

Introduction

The computer-assisted telesurgery robot system can eliminate geographical constraint. The doctor is able to perform complex surgical manipulations in a remote room which is far away from the operating room (OR).^1–3 The idea of telesurgery robot can be dated back to the early 1970s, which is proposed for providing surgical care for astronauts. ⁴ However, the telesurgery has not been substantially translated into clinical use currently, although the famous transcontinental telesurgery has been performed in 2001.^5,6 One of the main barriers is that the telesurgery system requires dedicated network with low latency and reliable data transmission, ⁷ which was too expensive for the general hospitals to afford. With the progress of communication technology in recent years, high-speed data transportation over the Internet is becoming cheaper and cheaper, which makes is possible to translate the telesurgery into clinic. The telesurgery system requires long-distant telecommunication with not only time delay as small as possible but also stable transmission. However, the delay period of data transmission over the Internet medium is based on the store-and-forward mode, while the delay time depends on the congestion, bandwidth, distance, and so on. ⁸ The time-varying delay is one of the prime factors that can degrade the performance or even cause instability for the telesurgery system or medical robot,^9–11 which can be very dangerous for the patient. Many researchers have done a lot of work to overcome the barrier of the time-varying delay.^10,12,13 Many of them focus their researches on the bilateral control system with the Internet-based transmission, based on the dynamics of master–slave robot, ¹⁴ and some of them focus on real-time navigation based on continuum robots.^15,16

The telesurgery robot system will have a far-reaching practical significance. First of all, limited by the geographic and economic development in different regions and countries, the patient could not acquire equal medical resources. Remote surgery can save time and cost and also enable people in the backward areas to enjoy the better or the latest medical technologies. Second, with the telesurgery robot, surgical treatment can also be available for the wounded on the battlefield and in disaster areas. ¹⁷

Based on the self-designed “MicroHand S” minimally invasive surgery robotic system, ¹⁸ the robot remote control platform was developed. Figure 1 shows the structure of the telesurgery system. “MicroHand S” minimally invasive surgery system consists of the master hand, slave robot, and the stereo image system. In the remote control system, the “MicroHand S” robot keeps essentially the same platform. Based on Transmission Control Protocol/Internet Protocol (TCP/IP), a telesurgery communication protocol was set up. The stereo images of laparoscopy were transmitted by the hardware-based H.264 encoder/decoder. The medical robot laboratory of Tianjin University and the visual surgery laboratory of the PLA Rocket Force General Hospital, which were about 150 km apart, were built as the telesurgical demonstration environment. After phantom training and testing, a remote gall bladder removal surgery was performed.

Figure 1.

The telesurgery robot system based on “MicroHand S.”

Introduction to the “MicroHand S” system

The “MicroHand S” robot is composed of a surgeon console and a slave cart, as shown in Figures 2 and 3. The surgeon console of the system includes the image display system, control panel, foot pedals, and two master operating manipulators. The stereo image display system is used to provide the operation field of the surgical site to the doctor. The control panel is used to set the performance parameters of the system. The two master operating manipulators are the signal acquisition and output device of the surgeon’s operation. The foot pedals arranged in the surgeon console are to engage or disengage the instrument motion of slave cart, to engage or disengage the laparoscope’s motion, and to start electrocoagulation and cauterization.

Figure 2.

The surgeon console of the robot.

Figure 3.

The slave cart of the robot.

The slave cart consists of three operating arms, two of which are used to operate the surgical instruments and one to support the motion of laparoscope. These three arms are installed on a swivel head which connected the base of slave cart with a passive arm. The swivel can rotate around the vertical axis in two directions within the motion range [−90°, 90°] and move in the forward and backward directions. The passive arm can slide in an up-down direction during the pre-operative setup procedure.

The 4-degree-of-freedom (4-DOF) instrument of “MicroHand S” robot is of a roll-pitch-roll form as shown in Figure 4. The diameter of inserted part of the instrument is 8 mm, and the weight is 165 g.

Figure 4.

The 4-DOF minimally invasive surgery instrument for robot.

The stereo image system of “MicroHand S” includes the stereo laparoscope, image processor, and stereo image viewer on the surgeon console, as shown in Figure 5. The stereo laparoscope has dual optical system to simulate binocular vision of human. The image processor captures and enhances the dual images and then shows the two 1920 × 1080 resolution images on the stereo image viewer.

Figure 5.

Stereo image system.

Using “MicroHand S” system, the operator controls the 7-DOF master console and the movement is mapped to the slave side. For the master–slave position mapping, according to the lever effect caused by the incision point, kinematics of master manipulators and inverse kinematics were built in the Cartesian coordinates. For orientation mapping, the one-to-one joint correspondence was adopted to simplify the model of kinematics and reduce the time delay of operation as shown in Figure 6.

Figure 6.

Master–slave mapping scheme of surgery robot system.

Telesurgery control system

Main control unit

To realize the telesurgery communication, two workstations were set in the local and remote terminals, respectively. The workstations were linked through the network, and each of them connected with a robot controller at the master and slave sides to form the upper and lower machine control modes. The workstations were used to receive and process the data from the robot controller, coding information, and send or receive data between the remote workstations. The robot controller did the data collection, processing and computation, and the servo control of each joint of the master or slave robot.

Communication method

The control data of robot include the position and orientation (PO) signals of master manipulator, the feedback signal of slave robot, and those of the control panel and foot pedals. TCP was used to transmit control data through the network to ensure accuracy. The data package was 52-bytes long as shown in Table 1. In the data package, 40 bytes of PO structure contains the PO information of master or slave robot in floating point format; 4 bytes of input and output (IO) code encode the control panel and foot pedals in ushort point format; and 1 byte of Flag code in uchar point format defines source of the data package.

Table 1.

Data package format.

2 bytes	1 byte	1 byte	40 bytes	4 bytes	2 bytes	2 bytes
Package header	Flag code	Package length	PO structure	IO code	CRC	End of package

CRC: cyclic redundancy check.

In order to ensure the reliability of data transmission of the robot, the data packages need to be validated. In this article, the workstations of master and slave robots checked four parts of the data package: package header, package length, cyclic redundancy check (CRC), and end of package. Package header, package length, and the end of package can verify the independence and integrity of data package. The CRC was used to detect common errors caused by noise in transmission channels. If a data package was lost or a transmission error occurred, the workstation may deny command and discard package to prevent unexpected motion of the robot. The data package was sent every 10 ms, and the control loop of surgical robot for telesurgery was set as 50 Hz. Therefore, limited data lost would not make the system go out of control.

Image feedback system

Besides the control data, video and audio data including laparoscope images, monitoring images of the OR, and speech interaction need to be transmitted remotely. The video and audio data require much wider network bandwidth compared with control data. To ensure stability and transmission efficiency, the video and audio data are transmitted separately from the robot control data. A real-time streaming device (TVI-4000, Topvideo, Beijing, China) was chosen to encode video streams into the H.264 AVC format. The TVI-4000 supports one way voice and one way high definition image compression with low encoding/decoding delay, and has the ability of video transmission over IP-based network.

In this article, two TVI-4000 were used, one for laparoscope images, one for monitoring images of OR. For lightening the network burden, images of left and right channels of the stereo laparoscope were merged into a single picture. In the telesurgery, the viewer console of surgeon side was replaced with a three-dimensional (3D) monitor which works on the horizontal interlaced stereo display mode. The 3D monitor displays left and right images line by line and half the vertical resolution of each image, and as a result, the two images were compressed into 1920 × 540 resolution and merged in a 1920 × 1080 top-bottom frame.

In telesurgery, the surgeon needs voice communication to give instructions to the local assistants and nurses, thus microphones and speakers were added to both local and remote sides. In conclusion, the whole image feedback system includes the stereo laparoscope system, OR monitoring system, two TVI-4000 systems, microphones, and speakers as shown in Figure 7.

Figure 7.

Image feedback system.

Experiment and result

Network environment test

In order to get the parameters of the network environment between the surgeon console and slave cart, a self-designed program for testing the network environment was developed and deployed in the server and client sides. The data packets we used had the same format and capacity as the actual control signals, while, meantime, a series of numbers representing the sending order were compiled in them. The checking mechanism was also used in the testing program, and the data packets with errors that were checked out would be discarded directly. In the test, the data packets were sent at a consistent frequency, and the data numbers and sending time were recorded. While data packets were received, they would be checked and then answered back if there was no packet loss. The data number and the receiving time were recorded if the data were checked to be right. Thereby, with the same numbers, the bidirectional delay of data packet transmission was defined by the difference in sending and receiving time. And, the packet loss ratio could be calculated by the amount of transmitted data packets and answered data packets.

In this article, for obtaining the fluctuations of network environment in one day, the test was done 10 times in the same day. The total amount of data packets is over 30,000 in each test, and the transmission cycle was 20 ms. In total, 10 experiments were done in different periods (1 h was defined as a period) during a day. It can be assumed that the testing results over 10–20 min could represent the fluctuations in each period. Table 2 shows the testing conditions and results, and the delay of data packets transmitting ranged from 9.1 to 24.6 ms. The minimum delay was stable, which was about 4–5 ms. On the contrary, the maximum delay of each test experienced obvious fluctuations. The maximum one was 408 ms (at the fourth time, 11:25–11:38 a.m.). And, during the 10 times of experiments, the packet loss was inevitable; however, the packet loss rate was relatively small, whose maximum value was less than 4.5%.

Table 2.

Internet environment from Tianjin to Beijing in one day.

No.	Date/time	Minimum delay (ms)	Maximum delay (ms)	Average delay (ms)	Amount of packages	Packages loss	Loss rate (%)
1	28 August 2014 8:45–8:56	5	331	22.5	32,912	310	0.94
2	28 August 2014 9:30–9:41	4	360	24.4	30,993	1272	4.10
3	28 August 2014 10:29–10:41	4	388	23.8	33,257	490	1.47
4	28 August 2014 11:25–11:38	4	408	17.7	37,146	436	1.17
5	28 August 2014 12:49–13:00	4	333	22.6	30,611	300	0.98
6	28 August 2014 13:32–13:54	4	392	24.3	65,999	461	0.70
7	28 August 2014 14:40–14:52	4	329	14.0	36,394	341	0.94
8	28 August 2014 15:26–15:37	5	263	24.6	32,224	41	0.13
9	28 August 2014 16:20–16:32	4	329	9.1	34,883	83	0.24
10	28 August 2014 17:30–15:43	4	206	22.1	37,684	690	1.83

Video transmission latency

Video transmission latency includes the overall delay of video capture, encoding, transmission, decoding, and display. In telesurgery system, the fluctuation of image feedback latency would prolong response time of operation and increase surgical risk. Quantitative measurement of image transmission delay would help to evaluate the total delay of the system and analyze the feasibility of telesurgery.

To measure the image transmission delay of telesurgery, a dual-clock method was used in this article. Two millisecond meters were located at the surgeon console and slave cart of the robot, respectively, and both of the meters could be activated by a self-designed program running on the workstation at slave cart. Before sending the active signal, the program would ping the remote workstation for 10 times and measure an average delay of command signal. Then, the program would activate the local and remote millisecond meters successively with this delay time for synchronous starting. In the experiment, the millisecond meter at slave cart was filmed by the laparoscope and transmitted to the surgeon console. A high-speed camera was used to shoot the remote laparoscope image and the local millisecond meter together. The video transmission latency could be measured with the time difference of the two millisecond meters recorded by high-speed camera, as shown in Figure 8.

Figure 8.

Experiment of image feedback delay: (a) raw laparoscope image, (b) two-dimensional laparoscopic remote image, and (c) 3D laparoscopic remote image.

The latency of raw laparoscope images that were not transmitted, the two-dimensional (2D) laparoscopic remote images, and the 3D laparoscopic remote images were measured. The 3D images were transmitted as top-bottom frames and merged to horizontal interlaced line-by-line images at the surgeon console. Experiment results are shown in Figure 9. The laparoscope images had an inherent delay of about 120 ms, which was caused by video capturing, image enhancing, and displaying. The value of image transmission delay fluctuated within about 60–80 ms. The average delay time was 296 ms for 3D image and 274 ms for 2D. The latencies of 2D and 3D images transmission showed little difference, because they had the same resolution and consumed similar bandwidth. Therefore, the 3D laparoscope images could be used in the telesurgery without introducing too much delay.

Figure 9.

Analysis of image feedback delay.

Phantom tasks

To train the surgeon and validate the telesurgery system, surgical phantom tasks were performed. In the experiment, a phantom was placed in a laparoscopic surgery operation training box. The box was placed on a table beside the slave cart of robot. The stereo laparoscope and two surgical instruments of robot were inserted into the box through the holes on it.

Three surgeons and five students participated in the task. All of them had the experience of using local surgical robot system. After half an hour of remote operation training, the participator operated master manipulators in front of the master console and completed a pegboard task and a knot-tying task in the training box by remotely controlling surgical instruments under the visual feedback of laparoscope. Each task was repeated three times, and the time took by each task was recorded.

To complete the remote pegboard task, the operator shall control the left surgical instrument to pick up one rubber ring from a plastic column, then pass it to the right instrument with good coordination of the two hands, and finally, put the ring on another plastic column with the right instrument. After that, the operator shall repeat the process to move another rubber ring from the left to the right. The whole process is shown in Figure 10. The experiment involves basic functions such as picking, moving, exchanging, and placing.

Figure 10.

Remote pegboard task.

The remote knot-tying task requires the operator to tie the sutures with a knot using the surgical instruments as shown in Figure 11. This task involves flexible manipulation and multi-axis coordination of the telesurgery control system.

Figure 11.

Remote knot-tying task.

The pegboard and knot-tying tasks were progressing well. It took 35 s in average to move the rubber ring without fault, and the average knotting time was less than 8 s. The experiments showed good consistency of master–slave mapping, smooth and steady master–slave tracking, no erroneous operation, sensitive control of the instrument’s DOFs including rotation, deflection and open-close, and the flexibility satisfying requirements of the operator.

With increasing remote operation and training time, the performance will be improved. The experimental results show that the developed remote surgical robot system has good master–slave tracking performance and manipulability.

Animal task

All experimental animal procedures were approved by the Guidelines for the Care and Use of Laboratory Animals of Chinese Animal Welfare Committee. The surgeon console and the slave cart of robot was 120 km apart as shown in Figures 12 and 13. The connection from local to remote was based on the Internet with the 10-Mbit upload and download network bandwidth. The Internet connection was tested before the experiment. After successful anesthesia and intubation, a 18.5-kg sow was placed in the supine position. After setting up the slave cart of robot, the remote gall bladder removal surgery was begun. The surgeon has the experience of local surgical robot manipulation and passed 1 h teleoperation phantom training. The ratio of surgeon’s movement and the instrument tip was downscaled by a factor of 1/5 to reduce the effect of time delay and network packet loss.

Figure 12.

The laboratory room at Tianjin University, Tianjin, China.

Figure 13.

OR at the PLA Rocket Force General Hospital, Beijing, China.

The remote gall bladder removal surgery was successfully completed using the telesurgery control system. The preparation took about 2 h, including network test, teleoperation test run, animal anesthesia and intubation, and surgery robot placement. The telesurgery started at 10:57 and finished at 11:46. The operation time was about 50 min. Figure 14 shows the removed gall bladder. The blood loss was evaluated to be less than 10 mL, and no organ damage occurred. From the result shown in Table 2, the animal experiment passed the worst Internet environment in one day. The overall delay including controlling and image transforming during the operation was 253 ms, and the robot run stably. Compared with previous local animal study of “MicroHand S” robot, ¹⁶ the operation time increased for about 15 min. The main reason of time extension was downscaled movement ratio. However, the downscaled motion and extended period are acceptable according to an interview with the main surgeon.

Figure 14.

Removed gall bladder.

Conclusion

In this article, we have presented an integrated solution for the computer-assisted telesurgery system based on Internet. Compared with previous study^5,6 which was based on the high-speed terrestrial network with asynchronous transfer mode (ATM) service, our prototype based on TCP/IP is more affordable and widespread. The self-designed program for testing the network environment was developed to get the parameters of the network environment, and video transmission latency was measured with the time difference of the two millisecond meters recorded by the high-speed camera. Experiments based on a self-designed minimally invasive surgery robot “MicroHand S” were done to verify the telesurgery control system, and a gall bladder removal surgery of a sow was successfully performed between Tianjin and Beijing of China.