Sage Journals: Discover world-class research

Abstract

Master node delay is an unavoidable factor in a networked control system (NCS), which will lead to performance deterioration of the system, and large delay may even cause instability. In order to measure the master node delay in NCSs, its influence factors are analyzed and a non-perturbative black-box measurement method is proposed. In this method, a FPGA-based (Field-Programmable Gate Array) measurement device is developed, which sends out testing frames to master node periodically and triggers the master node to send out a response frame for every testing frame. Then the measurement device can get the measurement delay value by calculating the time interval between sending the testing frame and receiving the response frame, which consists of master node delay and some other delay that can be calculated precisely. Therefore, the accurate master node delay can be obtained by further processing the measurement delay value. A set of experiments were carried out and the results show that the proposed method can effectively measure the master node delay without a thorough understanding about its hardware and operating system software, and is not constrained by the type of operating system. Besides, as the master node delay can be measured exactly in the actual working condition without perturbing its user program by using this method, the results can reflect the real-time performance of the master node accurately. So that, it can provide a direct reference for choosing an appropriate master node for NCSs.

Keywords

the master node delay non-perturbative measurement networked control system real time ethernet

Introduction

Networked control system (NCS) is a kind of spatially distributed system wherein the control loop is closed through communication network.¹ NCSs can break the limit of point-to-point control and achieve remote operations, which makes it possible to design large-scale systems.² Compared to conventional control systems, NCSs have advantages in low cost of installation, flexibility in system implementation, ease of maintenance, efficient resource allocation and so on. However, the network delay is inevitable when control and status data are exchanged between master and slave nodes, which will degrade the performance of control systems and even cause system instability.³

There are two main types of network delay in a NCS.⁴ One is commonly called the network-induced delay, which can be divided into the propagation delay through the network medium and the transmission delay of the packets bits from the queue and into the link. The other is the processing delay of each component in NCSs due to its limited processing speed, such as the master node delay. Figure 1 shows a typical NCS model with the network delay.⁵ Where, τ _sc refers to the

{\begin{array}{l} x [(k + 1) T] = e^{A T} x (k T) + u (k T) \int_{0}^{T - τ} e^{A t} B d t + u (k T - T) \int_{T - τ}^{T} e^{A t} B d t \\ y (k T) = C x (k T) \end{array}

(1)

network induced delay from sensor to controller, τ _ca refers to the network induced delay from controller to actuator. The master node delay τ _c is the total time consumed by the master controller between receiving data from sensor and sending out control data to actuator. All these delays are involved in the control loop, when the total delay is smaller than control cycle, the state space equation of NCS is shown in equation (1), where τ is the total delay in NCS including τ _c, τ _sc and τ _ca. It can be seen that these delays will degrade dynamic performance and the stability of NCS, and add lots of difficulties in the development of high performance NCS.

Figure 1.

Block diagram of a discrete-time model of NCS.

In traditional communication networks which are not real-time, the network-induced delay is much larger than the master node delay because of the data collision, transmission rate and so on. As a result, the previous literature, whether review articles, or research papers about the stability of NCS,^3,6 linear system,⁷ nonlinear system⁸ and software defined network real-time system,⁹ mainly focused on τ _sc and τ _ca.

However, data collision of real-time Ethernet (RTE), such as EtherCAT, PROFINET IRT, Ethernet Powerlink, etc. Can be avoided,^10,11 and their bandwidth is as high as 100Mbps or even 1000Mbps, so the network-induced delay τ _sc and τ _ca have been greatly reduced. Besides, these delays can be calculated accurately based on the frame length and network topology. Some works^12-15 about the performance comparison of different RTEs provided the calculation methods of the minimum communication cycle of mainstream RTEs in common scenarios.

In fact, standard personal computer (PC) with real-time operating system (RTOS) as the master node is becoming more and more popular for its high computing ability, powerful integrated development environment and friendly human-machine interaction. For example, Windows operating system, equipped with a real-time extension software such as TwinCAT,¹⁶ Codesys¹⁷ or Kithara,¹⁸ is competent to be the master node of a NCS. As the RTOS on standard PC is responsible for multi task, the master node delay is inevitable, and even larger than the network-induced delay. Hence, the master node delay τ _c will account for an increasing proportion of network delay. Master node delay will affect the minimum communication cycle, dynamic performance and stability of NCSs, so analysis and measurement of the master node delay is needed, and a convenient and effective measurement method for the master node delay is proposed in this paper.

The paper is organized as follows. In Section 2, the factors causing the delay of PC-based master node are analyzed. Section 3 introduces the details of the proposed measurement method. Then, the development of the measurement device is given in Section 4. Section 5 is a set of experiments which give a performance comparison of different master nodes. At last, a conclusion is presented in Section 6.

Analysis for master node delay

The master node usually works as the controller in NCSs, while the sensors and actuators are all slave nodes. Hence, the master node delay should be the duration between the master node receiving an upstream status frame and sending out a downstream control frame.

As shown in Figure 2, the master node delay is determined by both hardware and software, of which hardware delay is influenced by both CPU architecture and Design of motherboards and software delay includes operating system (OS) delay and algorithms efficiency. The OS delay can be further divided into Interrupt delay, Preemption delay, Task switching delay, Signal alternating delay, Deadlock release delay and so on.

Figure 2.

Factors of master node delay in NCS.

There are software and hardware methods for measuring OS delay. The software methods mainly employ code instrumentation¹⁹ or embedded software testing tool.²⁰ The accuracy of these methods is not very high, because the operating system computational resources are occupied by both application program and testing software, and the precision of system clock is limited. Besides, a thorough understanding on OS principles is vital to modify system-level code.

The hardware methods^21-23 usually employ a signal generator to trigger the master node under test to produce a single output, and then an oscilloscope is used to measure the response delay between the input and output signals. Although the measurement result of hardware method is more accurate than that of software, it is not convenient for statistical analysis and long-term measurement.

However, all the methods above measure the real-time performance factors separately, and the actual delay of the master node is a combination of coupling factors. Besides, the master node delay caused by hardware is difficult to quantify independently. So it is difficult to obtain accurate master node delay with a white-box measurement method, and a non-perturbative black-box measurement method is proposed in this paper.

Measurement method for the master node delay time

In order to make the measurement result close to the actual working condition as more as possible, a NCS with master-slave structure is adopted, as shown in Figure 3. Where, a standard PC with RTOS or non RTOS works as master node under the test, and the measurement device based on FPGA works as a slave node. The measurement process is implemented in a FPGA chip, where high-speed clock and hardware solution can ensure the accuracy of measurement results.

Figure 3.

Overall schematic of measurement method.

The proposed measurement method regards the master node as a black box, and the measurement device periodically sends upstream frames to the master node and receives the downstream response frames. At the same time, a high-precision timer designed in the FPGA records the sending and receiving time, so that the master node delay can be calculated. Acting as a trigger event, receiving an upstream frame makes the master node to implement data reception, data analysis, computational algorithm, downstream frame sending, etc. The whole process not only involves the delays caused by OS and computational algorithm, but also includes hardware factors. As a result, the master node delay of each communication cycle can be measured with high precision. Besides, the proposed method does not require any extra underlying system operation, which greatly reduces the difficulty of measurement and can be easily used in different types of OS.

Measurement Process

As shown in Figure 4, T _{measure_k-1} and T _{measure_k} denote the measurement delay of cycle k-1 and cycle k, respectively. T _{master_k-1} and T _{master_k} denote the master node delay of cycle k-1 and cycle k, respectively. T _cycle denotes the communication cycle. The measurement process is shown in Figure 5. A detailed description of the measurement process is is as follows:

a. At the beginning, set T _cycle as the communication cycle of this time of measurement and set T _{measure_0} as the initial value 0 which will not affect the statistical results. Then the master node is responsible for initializing the communication.

b. After initialization, the measurement device switches to cycle communication state. In this stage, the measurement device will send an upstream frame with the measurement result of last communication cycle T _{measure_k-1} cyclically according to the set period, and start an internal high-precision timer when sending out the upstream frame.

c. After receiving the upstream frame, the master node parses T _{measure_k-1}, calculates the master node delay T _{master_k-1}, and writes the value into a file for further analysis. At the same time, the master node sends a downstream frame to the measurement device to response the upstream frame.

d. The measurement device parses the received downstream frame online, and latches its internal high precision timer at the time of receiving the downstream frame. So that the measurement delay T _{measure_k} of this cycle can be obtained.

e. Repeat steps (b) – (d) to collect sufficient data for statistical analysis.

Figure 4.

Sequence diagram of measurement process.

Figure 5.

Measurement process.

This method is mainly aimed at standard PC-based master node, which has powerful computing capability. Therefore, the measurement results are transmitted to the master node for subsequent calculation and data statistics just as step (c) shown.

The master node delay algorithm

The direct measurement result T _{measure_k} contains the propagation delay, transmission delay and the device delay of the slave node, so subsequent operation is needed to calculate the exact master node delay, that is, the master node delay algorithm shown in Figure 6.

Figure 6.

Master node delay algorithm.

As shown in Figure 4, T _{master_k} is the master node delay in cycle k

T_{master_k} = T_{measure_k} - D_{t x} - D_{r x} - T_{pro} * 2

(2)

Where, T _{measure_k} is the direct measurement delay time value measured by the measurement device in a communication cycle, which will be encapsulated into the upstream frame and sent to the master node. T _pro is the time delay caused by the propagation of data frames over the network, which is mainly related to the length of the network cable L _cable. The propagation delay is 4.8∼5.3 ns/m for category 5 cable. Then

T_{pro} = L_{cable} \times 5 n s

(3)

D _tx and D _rx represent the delays caused by the measurement device when sending the upstream frame and receiving the downstream frame, respectively. The delays mainly consist of the delay of physical layer chip, D _{PHY_tx} and D _{PHY_rx}, and the delay of data frame transmission, T _{frame_tx} and T _{frame_rx}. Different physical layer chips have different D _{PHY_tx} and D _{PHY_rx}, which can be obtained from its datasheets or through experimental testing. Then

D_{t x} = D_{PHY_t x} + T_{frame_t x}

(4)

D_{r x} = D_{PHY_r x} + T_{frame_r x}

(5)

The network under test is the Fast Industrial Ethernet whose bandwidth is 100Mbps, so the transmitting rate is 80ns/byte. According to the frame structure of standard Ethernet shown in Figure 7, both upstream and downstream data frames take the Data and Pad region as the minimum value 46 bytes. Then, the length of data frame L _frame and the delay of data frame transmission T _frame can be obtained.

\begin{matrix} L_{frame} = L_{Pre} + L_{SFD} + L_{DestMAC} + L_{SourceMAC} + L_{Type} + L_{Data} + L_{FCS} \\ = 7 + 1 + 6 + 6 + 2 + 46 + 4 \\ = 72 byte \end{matrix}

(6)

\begin{matrix} T_{frame} = L_{frame} \times 80 ns / byte \\ = 72 byte \times 80 ns / byte \\ = 5760 ns \end{matrix}

(7)

Figure 7.

Frame structure according to IEEE802.3.

According to formula (4) – (7), both T _{frame_tx} and T _{frame_rx} can be calculated to be 5760ns, and the sum is 11,520ns. Besides, T _{master_k} can be calculated by combining the formula (2) – (7).

T_{master_k} = T_{measure_k} - 2 * L_{cable} * 5 - D_{PHY_t x} - D_{PHY_r x} - 11520 n s

(8)

Development of measurement device

The proposed measurement method in this paper adopts the master-slave structure. The slave node is developed based on a FPGA chip, whose hard real-time property can realize high precision measurement. For the master node, a program is necessary for standard PC with RTOS or non RTOS to response the frame from slave nodes.

Program Architecture of Master Node

The program running on master node is developed with C++, which consists of two parts: communication driver and control task program, as shown in Figure 8, communication driver and control task are two threads started by the master node. They cooperate with each other to implement the tasks of receiving and sending frames and processing measured values through events and semaphores.

Figure 8.

Program architecture of master node.

The control task is responsible for the parse of upstream frames, the storage of measured values and the packaging of downstream frames. In practice, all computational algorithms will be running on the master node under test, if which can be added to the program, the measurement results can be closer to the actual situation and more referential.

The communication driver is responsible for receiving and sending data frame, controlling of communication state machine and driving network card. In this part, communication function is realized by calling system API functions provided by RTOS employed by the master node under test. With this modular software design, the method can be implemented on different types of OS with only little extra work.

Development of measurement device node

The measurement device works as a slave node in NCSs, which is developed based on a FPGA chip from Altera, whose advantages are hard real-time, flexible, high-speed and reconfigurable. The device not only acts as a slave node to communicate with the master node, but also can obtain nanosecond resolution results with the high-precision timer in FPGA. Besides, the Nios II soft core technology is used to facilitate the analysis of the downstream frame and the preparation of the upstream frame.

The module of the FPGA program is described in detail as shown in Figure 9:

Figure 9.

Design of FPGA module.

a. High precision timer in the Delay measurement module will be started when FPGA sends an upstream frame and latched when FPGA receives a response downstream frame from master node. The value of the timer will be sent to Nios II CPU.

b. Nios II CPU encapsulates the Delay value into upstream frame when its value is updated cyclically.

c. Cycle load module generates a signal with a set time interval to trigger the Uptransfer module to send upstream frame with the measured value to master node.

Experiment

In order to verify the proposed method, a set of experiments are designed to compare the master node delay of three different RTOSs (Windows 7 + Kithara, Windows CE 6.0 and Linux with RT-PREEMPT patch). Besides, a master node with Windows 7 is adopted as a reference for conventional non-real-time operating system. At the same time, industrial personal computer (IPC) and laptop are employed to compare the influence of hardware platform on the master node delay.

Experimental environment building

As shown in Figure 10, the hardware platforms of the master nodes are two IPCs and one laptop Thinkpad E430C (abbreviated as IPC1, IPC2, laptop) whose hardware information is shown in Table 1.

Figure 10.

Experimental environment.

Table 1.

Hardware information of master node.

	CPU	CPU frequency, GHz	Memory
IPC1	Intel celeron J1900	1.99	DDR3 2 GB
IPC2	Intel atom N270	1.60	DDR2 1 GB
Laptop	Intel core i5-2520M	2.50	DDR3 4 GB

As shown in Table 2, different OSs and communication drivers have been employed, where communication driver is an important uncertain factor can influence the master node delay.

Table 2.

Table of software experimental conditions.

No.	Hardware platform	Operating system	Driver
1	Laptop	Win7, 32bits	NDIS driver
2	Laptop	Ubuntu14.04.5 Desktop (32bits) +RT3.18.59	Libcap
3	IPC1	Ubuntu14.04.5 Desktop (32bits) +RT3.18.59	Libcap
4	IPC1	Ubuntu14.04.5 Desktop (32bits) +RT4.1.40	Libcap
5	IPC1	Ubuntu14.04.5 Server (32bits) +RT4.1.40	Libcap
6	IPC2	WinCE6.0	Miniport driver
7	Laptop	Win7, 32bits + kithara	KRTS packet
8	IPC1	Win7, 32bits + kithara	KRTS packet

Some cross-comparison of the measurement results on these platforms are provided to show the measuring performance of the proposed method.

The right side of each image in Figure 10 is the measurement device, whose FPGA chip is EP4CE10F17C8N from Altera and PHY chip is LAN8710A from SMSC. According to LAN8710A’s datasheet, the total sending and receiving delay of LAN8710A is 320ns.

D_{PHY_r x} + D_{PHY_t x} = 320 ns

(9)

The master nodes under test are connected to the measurement device with a CAT-5 cable, and the cable length L _cable is 1m.

T _{master_k} can be calculated with formula (8).

T_{master_k} = T_{measure_k} - 11850 ns

(10)

Eight different experimental conditions as shown in Table 2 were established, and each measurement lasted for about 40 min with the communication period of 4 milliseconds. There are about 600,000 delay values collected for every time of measurement.

Analysis of experimental results

The histograms of these eight measurements results are shown in Figure 11. The X-axis is the value of the master node delay and the Y-axis is the percentage. Because there are lots of uncertain factors for master node delay, so the tested results are random, which may be particularly high for the poor real-time performance. As a result, the master node delay is widely distributed on the X-axis. There are always some long-time delays of small proportion in the overall sample, which results in a big blank on the right side of the original diagram. Thus, in order to show the results clearly, the data-intensive distribution areas are intercepted in Figure 11.

Figure 11.

Histogram of experiments data.

As shown in Table 3, the maximum delay time is one of the most important indicators in practice, which determines the minimum control cycle of NCSs. Too large delay means the long control period, which is intolerable in high-precision control systems. The mean value and standard deviation synthetically reflect the overall delay and data consistency of the experimental samples. The standard deviation also reflects the jitter of the PC-based master node. When the maximum delay value meets the requirement, they can help to choose more stable master node. The last column shows the ratio of data excessive 1 millisecond, which is usually set as the communication cycle in the field of industrial control.

Table 3.

Statistic of experiments data.

No.	Max/ns	Mean/ns	StDev	Ratio of data excessive 1 ms
No.	Max/ns	Mean/ns	StDev	Num	Ratio
(1)	197,811,830	400,410	1,177,048	1267	0.210%
(2)	13,587,570	274,455	72,309	246	0.041%
(3)	8,976,970	127,327	77,323	362	0.060%
(4)	5,735,890	108,176	44,683	153	0.025%
(5)	4,946,130	110,120	19,382	22	0.004%
(6)	770,770	234,614	17,972	0	0
(7)	138,790	107,872	2852	0	0
(8)	116,030	48,428	7935	0	0

For Windows 7, the maximum delay is about 200 ms and the values are scattered, which means it cannot meet the real-time requirement for a master node in NCSs.

For RT Ubuntu with a RT-PREEMPT patch, the maximum delay is much smaller than Windows 7. However, it is still in millisecond level and the jitter is high. It is supposed the communication driver reduces its real-time performance, which limits the minimum communication cycle.

For WinCE 6.0, although the mean delay is larger than RT Ubuntu, its maximum is about 700 microseconds, which is better and conforms to the actual situation.

For Windows + Kithara, the maximum delay is about 100 microseconds, which has the best real-time performance. The standard deviation is much smaller than other OSs, which means the jitter of Windows + Kithara is also the best.

In addition, (2), (3) and (7), (8) in Figure 11 can also show the influence of hardware platform to the master node delay.

Conclusions

In NCSs, master node delay is inevitable, especially for the PC-based control system. This study analyzed the factors lead to the master node delay in NCSs, and proposed a non-perturbative black-box measurement method. As the delay measurement is implemented on a FPGA-based slave node, the master node can be tested without perturbation and all the complex factors of master node are consisted in the measured value as a black box, so that high accuracy can be achieved.

A set of experiments have been conducted, where different master nodes with different OSs were employed to verify the proposed method. And a cross-comparison of the measurement results have been concluded. Therefore, besides the master node delay measurement method, this paper also gives a reference for selecting an appropriate master node of NCSs.

Footnotes

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the National Key R&D Program of China under Grant 2019YFB2006200.

ORCID iD

Yongmeng Huang

References

You

K-Y

Xie

L-H

. Survey of recent progress in networked control systems. Acta Automatica Sinica 2013; 39(2): 101–117.

Gupta

Mo-Yuen Chow

. Networked control system: overview and research trends. IEEE Trans Ind Elect 2010; 57(7): 2527–2535.

Branicky

Phillips

Wei

. Stability of networked control systems: explicit analysis of delay. In: Proceedings of the 2000 American Control Conference. ACC (IEEE Cat. No.00CH36334), Chicago, IL, USA, 28–30 June 2000, pp.2352–2357.

Zhang

Gao

Kaynak

. Network-induced constraints in networked control systems-a survey. IEEE Trans Ind Inform 2013; 9(1): 403–416.

Chow

Tipsuwan

. Network-based control systems: a tutorial . In: IECON'01. 27th Annual Conference of the IEEE Industrial-Electronics Society (Cat. No.37243), Denver, CO, USA, 29 November-1 December 2001, pp.1593–1602.

Sun

X-M

Liu

G-P

, et al. Input-to-state stability for networked predictive control with random delays in both feedback and forward channels. IEEE Trans Ind Elect 2014; 61(7): 3519–3526.

Liu

, et al. Modeling and control of networked control systems with random network-induced delay and packet-dropout. In: 2017 12th IEEE Conference on Industrial Electronics and Applications(ICIEA), Siem Reap, Cambodia, 18–20 June 2017, pp.786–791.

Pang

Z-H

Liu

G-P

Zhou

, et al. Data-based predictive control for networked nonlinear systems with network-induced delay and packet dropout. IEEE Trans Ind Elect 2016; 63(2): 1249–1257.

Kumar

Hasan

Padhy

, et al. End-to-end network delay guarantees for real-time systems using SDN. In: 2017 IEEE Real-Time Systems Symposium (RTSS), Paris, France, 5–8 December 2017, pp.231–242.

10.

Wollschlaeger

Sauter

Jasperneite

. The Future of Industrial Communication: Automation Networks in the Era of the Internet of Things and Industry 4.0. IEEE Ind Elect Mag 2017; 11(1): 17–27.

11.

Jaspemeite

Schumacher

Weber

. Limits of increasing the performance of industrial Ethernet protocols. In: 2007 IEEE Conference on Emerging Technologies and Factory Automation (EFTA 2007), Patras, Greece, 25–28 September 2007, pp.17–24.

12.

Robert

Georges

Rondeau

, et al. Minimum cycle time analysis of Ethernet-based real-time protocols. Int J Comput Commun Control 2012; 7(4): 744–758.

13.

Seno

Vitturi

Zunino

. Real time Ethernet networks evaluation using performance indicators. In: 2009 IEEE Conference on Emerging Technologies & Factory Automation, Mallorca, Spain, 22–25 September 2009, pp.1–8.

14.

Cena

Seno

Valenzano

, et al. Performance analysis of Ethernet Powerlink networks for distributed control and automation systems. Comp Stand Inter 2009; 31(3): 566–572.

15.

Prytz

. A performance analysis of EtherCAT and PROFINET IRT. In: 2008 IEEE International Conference on Emerging Technologies and Factory Automation, Hamburg, Germany, 15–18 September 2008, pp.408–415.

16.

Bermudez-Ortega

Besada-Portas

Lopez-Orozco

, et al. Developing web & TwinCAT PLC-based remote control laboratories for modern web-browsers or mobile devices. In: 2016 IEEE Conference on Control Applications (CCA), Buenos Aires, Argentina, 19–22 September 2016, pp.810–815.

17.

Liu

Zhou

, et al. Design and application of a communication system based on Codesys. Adv Mater Res 2014; 1044-1045: 1113–1117.

18.

Kithara . Future-proof Software Architecture with Kithara RealTime Suite[EB/OL, 2018. -02-28)[2019-06-19], https://kithara.com/cn/downloads#whitepaper-slideshow.

19.

Shen

, et al. Research of the real-time performance of operating system. In: 2009 5th International Conference on Wireless Communications, Networking and Mobile Computing, Beijing, China, 24–26 September 2009, pp.1–4.

20.

Wang

Zhu

Wang

. The hardware probe connection research of CodeTEST based on ARM. In: 2010 International Forum on Information Technology and Applications, Kunming, China, 16–18 July 2010, pp.333–335.

21.

Aroca

Caurin

. A real time operating systems (RTOS) comparison. In: The 6th Work-shop on Operating System, Bento Gonalves, Brazil, 2009, pp.2441–2452.

22.

Barbalace

Luchetta

Manduchi

, et al. Performance comparison of VxWorks, Linux, RTAI, and Xenomai in a hard real-time application. IEEE TRANSACTIONS NUCLEAR SCIENCE 2008; 55(1): 435–439.

23.

Hambarde

Varma

Jha

. The survey of real time operating system: RTOS. In: 2014 International Conference on Electronic Systems, Signal Processing and Computing Technologies, Nagpur, India, 9–11 January 2014, pp.34–39.

Measurement of master node delay in networked control systems