Abstract
Wireless sensor networks have been widely studied for decades. There are many research areas that have been explored such as service availability and energy efficiency. Load balancing is one of the key techniques which can be used to solve the tradeoff between the both sides. In this paper, we propose a novel real-time load balancing technique which does not require the knowledge of static traffic demands and is based on control theory. We first formulate the packet forwarding problem into a classic control model and then derive the controller model. Then, we propose an algorithm to implement the controller and perform the simulation. The simulation results show that, our algorithm can balance the network traffic in real time and decreases maximum link utilization (MLU) sharply. Our algorithm reduces MLU by 78.4% compared to the shortest path first (SPF) routing and by 53.8% compared to another packet scheduling algorithm which was proposed in a previous work.
1. Introduction
Wireless sensor networks (WSNs) have been widely deployed in many industrial and military environments to inspect, detect, monitor, and track the targets according to specific application scenarios. There are many research areas that have been explored such as service availability, energy efficiency, routing scalability, and reliability [1, 2]. In WSNs, communication efficiency and power consumption are always the main obstacles to improving service availability and energy efficiency. The power consumption of node and link (i.e., the module which is used to establish and maintain the wireless channel) is highly related to the corresponding traffic volume; thus, one method to solve the tradeoff between communication efficiency and power consumption is to distribute the traffic uniformly across the network. For this purpose, many load balancing approaches have been proposed to address this issue.
He et al. [3] proposed a real-time communication protocol named SPEED, in which a stateless nondeterministic geographic forwarding (SNGF) algorithm was employed. SPEED can support soft real-time communication based on feedback control and SNGF. The control overhead and transient congestion are also well handled. However, SNGF forwards each stream among random concurrent paths. Thus, if only very few paths with the same cost are available between each pair of nodes, there will be very few paths for SNGF to choose. Actually, two paths with the same cost can rarely be found between two nodes. Puccinelli and Haenggi [4] proposed a cost-based routing protocol named Arbutus with a built-in load balancing strategy. Arbutus allows each node to choose a parent of a lower depth node as the route to sink such that the routing loops can be avoided during the construction of routes and the control overhead can also be greatly reduced. However, it is possible that some neighbors of a node may choose the same parent of their lower depth nodes. Thus, this “parent” may become the hot spot such that the traffic will be much higher on the links connected with the hot spot while lower on other links. Dai and Han [5] proposed a node-centric approach to reduce hot spots to achieve load balance in WSNs. The algorithm grows a spanning tree and then iteratively selects the branch with the lowest load to graft onto the unassigned border node which generates the highest load. This algorithm can balance the network traffic flexibly; however, it is an offline solution which cannot be performed in real time. Kacimi et al. [6] derived a load balance solution by formulizing the network lifetime maximization problem into a nonlinear programming. However, both the optimal and heuristic solutions are obtained based on the assumption of having the prerequisites of the network topology and static traffic demand distributions. Gupta and Younis [7] proposed a method to achieve load balance among all clusters each of which is composed by several nodes in WSN. The least-energy-constraint node in cluster acts as the cluster-head and performs as “gateway” to forward data to the base station. However, this method achieves load balance in an offline way which is based on the static traffic demand knowledge and topology information. Meanwhile, it only balances the load among different clusters but not all nodes. Lu and Wong [8] proposed a multipath protocol and an algorithm to achieve load balance in WSN. However, the aim of the approach is to reduce the energy consumption of each node other than the link utilization of wireless link. Meanwhile, the load balance ratio has to be determined prior to performing the proposed protocol in the network by solving an offline nonlinear programming. Some other works also explored this issue while the solutions were also obtained by offline algorithms based on the knowledge of the whole network topology and the accurate traffic demands [9].
Therefore, although the research on load balancing for WSNs is gaining more interest in recent years [1, 10, 11], to our knowledge, the issue is still open. In this paper, we solve the following problem. How to achieve real-time load balance in WSNs only based on the knowledge of local real-time traffic while the hot spots can also be avoided? To answer, we present a novel approach which contains two main contributions.
First, we propose a model in which the sending queue, receiving queue, and the processing unit of router are considered as one element depicted by control theory [12]. Then, we refer the different state parameters and their relationship as different variables and elements of the model. According to some reasonable assumptions, we model each router as one closed-loop system while the output of its controller is the total size of packets to be proceeded in some way, for example, to drop or to dispatch on other interfaces or even to forward as regular.
Second, we design an algorithm for the controller to schedule packets. It forwards the packets on different links depending on the current link utilization (LU).
Meanwhile, we simply discuss how to design a routing protocol to be compatible with our approach to ensure that no packet will be forwarded in a loop (e.g., named data network (NDN) can be employed [13, 14]).
There are two main merits of our approach. (1) There is no knowledge of the static accurate traffic demands needed for using in the model because the traffic can be balanced according to the real-time output of the controller. (2) The hot spot can hardly be formed because the algorithm balances the traffic by following the theory from which the controller is designed. The control theory ensures the actual LU to follow the expected one. Meanwhile, we also perform simulations to verify the effectiveness and performance of our approach versus other solutions.
The rest of the paper is organized as follows. Section 2 describes the control theory-based model and then the algorithm. Section 3 describes the simulation and analyzes the results. Section 4 concludes the paper and describes the future works.
2. Methodology
In this section, we first present the model and interpret how it works. Then, we describe the algorithm which is designed based on the controller. Finally, the employed routing protocol is also described.
2.1. Theoretical Model
The model is established for each router. The main performance of load balancing is maximum link utilization (MLU); thus, we set the aimed link utilization as the reference input of the model while the current forwarded packet size in a time unit is set as the output. Meanwhile, the difference between the reference input and the output of feedback is used as the input of the controller. The output of the controller is the way to schedule packets in the router. The feedback element is responsible for converting the real-time packet size into LU. Notation section shows the notations used in this paper. Figure 1 shows the block diagram of the model based on control theory [12].

The block diagram of the theoretical model based on control theory.
In notation section, each notation with variable s is the Laplace transformation [15] of the corresponding time domain variable. We can easily get the model from Figure 1, and it can be stated as follows:
In this model, the sending and receiving units, queues, and the processing unit are described as

The system diagram of
In (3),
According to [20],
Therefore, the controller
From (3)–(6), we can see that the system is critically stable (
2.2. Algorithm
We first describe our algorithm and interpret how it works. Then, we clarify why it can implement the expected control characteristic. Algorithm 1 shows the packet scheduling pseudocode.
// The function Schedule is invoked each time a packet is received. In the pseudocode, pkt is the current processing packet, pkt.len is the length of pkt, link.sizeInT is the total size of received packet in time interval T and to be forwarded on “link”, getCap(route) is used to get the link capacity of route, K is the constant reference LU and listLU is the list of LU of each link on the router. Input: pkt. Output: pkt, return status. Init(LU); //LU is init to 0 Init(lastTime); //lastTime is init to 0 Schedule(pkt) (1) (2) link.sizeInT += pkt.len; (3) (4) (5) trfRate = link.sizeInT/(curTime − lastTime); (6) (7) (8) (9) (10) (11) (12) (13) (14) lower depth of the current router (15) (16) otherLink.sizeInT += pkt.len; (17) link.sizeInT −= pkt.len; (18) (19) (20) Mark the last minLUas temporarily deleted from listLU; (21) (22) (23) (24) link.sizeInT −= pkt.len; (25) (26) (27)
The scheduling algorithm first updates the LU of the link on which the packet is to be forwarded according to the route. If the LU is lower than K, the packet will be forwarded normally. Otherwise, the packet will be forwarded on another link based on two conditions: (1) the LU of the other link is lower than the original one; (2) the router connected on the other link is not on the lower depth of the current router. If any of the above conditions is not satisfied, the packet will be dropped if it is a Transmission Control Protocol (TCP) packet or will be forwarded as usual if it is not TCP. Dropping TCP packets will not affect the integration and reliability of communication [21].
In the algorithm, a packet will be rerouted on another link when the LU of the current link is high. This is a way to slow down the increasing speed of LU on current link while speeding up that on the other link which owns lower LU. Hence, the characteristic of differentiating element with first-order inertia is implemented. This characteristic means that the response acts slower than the stimulation in some parts of the element while acting quicker in some other parts. Thus, the packet rerouting here changes the speed of LU varying and hence implements the characteristic. The characteristic of the first-order differentiating element is that the response reflects the gradient of the stimulation. Thus, it is implemented by dropping TCP packets. It is straightforward that dropping TCP packets will decrease the sending rate of source because of the flow control mechanism of TCP [21]. Hence, this actually decreases the packet arriving rate on the router in a short time in future. The proportional element is implemented by the case in which the packets are forwarded as usual. In the algorithm, only function “
2.3. Routing Protocol
From the statement 14 of the Algorithm 1, we can easily see that a routing protocol is needed to ensure the link, on which packet is rerouted, not to be connected with node which routes the packet to the current node again (directly or within some hops). Thus, in this paper, we design a routing protocol which includes two phases to ensure that each packet can be routed correctly in WSN. First, each router can obtain the next hop and path cost of each neighbor to the base station by exchanging the routing entries with its neighbors. Second, the neighbors which are closer, but not on the shortest path of current router to the base station, will be put in the candidate set of current router in which the routers are used to share traffic with the router that is on the shortest path. Once each router obtains the candidate set, Algorithm 1 can be performed each time a packet is received.
Generally, most traffic in WSN is generated by sensor nodes and is sent to the base station. Thus, each router only needs to flood its router ID (which can be generally derived in link state routing protocols, e.g., open shortest path first protocol, OSPF) and the routing entry which destines to the base station. When the information is derived in a router, it is flooded and cached in its neighbors and then will not be flooded again. Due to having the number of neighbors, each router can start to calculate the “candidate set” when all of the routing entries, which are destined to the base station, and router IDs of its neighbors are received. Each neighbor can be determined to be candidate or not by Algorithm 2.
// The function CndDt means “Candidate Determination” which is used by each router to derive the neighbors that can be used by Algorithm 1 to share traffic. In the pseudocode, rtEnt is the routing entry, which destines to the base station, of a neighbor; rID is the Router ID of the neighbor which owns rtEnt; rtb is the routing table of the current router; rtb.bs is the routing entry which destines to the base station in rtb; rtb.bs.nxhp is the IP address of the next hop; rtb.bs.cost is the path cost from the current router to the base station; rtEnt.nxhp is the IP address of the next hop; rtEnt.cost is the path cost from the router associated with rID to the base station. Input: rtEnt, rID. Output: rID. //which is in the candidate set CndDt(rtEnt, rID) (1) (2) current router on the shortest path to the base station. (3) (4) rID on the shortest path to the base station. (5) (6) the base station is lower than that from the neighbor rID to the base station. (7) (8) push(rID, CandidateSet); //Mark rID as the router which can be used by Algorithm 1 to share traffic that is forwarded from the current router to the base station. The rID is on the upper depth of the current router. That means, when a packet, that is destined to the base station, is forwarded from the current router to rID, it will not be routed back to the current router anyway if every router in the (9) (10)
3. Simulation
In this section, we first introduce the environment and perform the simulation in different scenarios to evaluate the effectiveness and performance of our approach. Then, we compare and analyze the result of our solution versus others.
In this paper, we use NS2 [22] to simulate all scenarios of which the parameters are shown in Tables 1 and 2.
Simulation scenarios.
Topology information.
There are four types of topologies in Table 1. The randsmall and randbig are pure-random graphs which are generated by GT-ITM [23]. The powsmall and powbig are Power-Law graphs which are generated by Inet-3.0 [24]. The two types of graphs are used to simulate different network structures in real world. The only difference between randsmall (powsmall) and randbig (powbig) is the router number which is used for simulating different size of network. Traffic Dist is used to simulate the different traffic source distribution in the real network. Uniform and Pareto are the most regular. Traffic Source Type is set to Constant Bit Rate (CBR) of which the burst interval is constant (e.g., audio). The rate of each source is set to 1 Mb/s and the maximum link capacity is set to 6 Mb/s. In the simulation, we perform our algorithm which is named “Control Theory” and other packet scheduling algorithms which are proposed or employed in our previous work [25]. NONE is not a scheduling algorithm in actual. It just routes packets by the shortest path first (SPF) policy. RR is the round robin scheduling algorithm which chooses the link in turn each time it forwards packet. In this paper, we only allow RR to choose the links which are not connected with lower depth nodes to avoid loops. SLUBP is the algorithm which only chooses the link with the lowest current LU. In SLUBP, only the source node performs the algorithm and encapsulates each packet. Other nodes can only decapsulate the packet and forward it as normal. All parameters in (10) are set in Table 3 for each simulation. MLU of all simulations is shown in Figures 3(a)–3(h).
The parameters in (10) for the simulation.

MLU in each topology and traffic distribution.
In each figure, the number of traffic demands is from 100 to 1200. Each simulation is performed one time at different number of traffic demands. In all figures, we can easily see that our proposed algorithm is much better than others. Regarding the same topology, each algorithm performs better when the traffic demands are allocated in uniform distribution. It is straightforward that Pareto distributed traffic demands have higher possibility to form the hot spot node. With the same distribution of traffic demands, it is not very distinct that the algorithms perform better in random topologies than in Pareto topologies. This is because most traffic in WSNs is finally sent to the sink, which is used to gather the sensing data. Thus, the biggest hub is the sink whenever there is any other hub in the topology or not. The MLU of network is generally determined by the traffic status on the links that is connected with the sink in most case. In other words, other hubs in the topology affect MLU very little. In some cases, we obtain lower MLU at higher number of traffic demands in each algorithm. This may be because some nodes are very far from others while being bound by many traffic demands such that higher link utilization may be raised on the links of these nodes. Compared to other algorithms, Control Theory performs more stably along with the increasing of the traffic demands. This proves that our algorithm can deal with large amount of traffic as well as the smaller ones.
On average of all simulations, compared to the regular SPF routing (i.e., NONE), Control Theory reduces MLU by 78.4%. Compared to SLUBP, Control Theory reduces MLU by 53.8%, while with nearly the same time complexity.
4. Conclusion and Future Works
WSNs have been widely studied for decades. There are many research areas that have been explored such as service availability and energy efficiency. Load balancing is one of the key techniques which can be used to solve the tradeoff between the performance and the network lifetime. In this paper, we have proposed a novel real-time load balancing technique which only uses the local real-time traffic information and is based on control theory. We have formulated the packet forwarding problem into a classic control model and have derived a controller model. Then, we have proposed an algorithm to implement the controller and have performed the simulation. The simulation results have shown that our algorithm can balance the network traffic in real time and decreases MLU sharply. Our algorithm has reduced MLU by 78.4% compared to the SPF routing and by 53.8% compared to SLUBP which was proposed in the previous work [25].
There is still much work to do in this topic. (1) To make sure the controller is stable, it is necessary to employ an artificial factor in the model to allow α to be any value greater than 0. Then, we need to design a new algorithm to implement the controller precisely and to achieve the stability depicted by the model. (2) Designing the specific routing protocol by which the packets distributed by our algorithm will not be routed in loop.
Footnotes
Notation
Conflict of Interests
The authors declare that there is no conflict of interests regarding the publication of this paper.
Acknowledgments
This research is partially supported by the National Basic Research Program of China (“973” Program) (2013CB329101), partially supported by the National Natural Science Foundation of China (NSFC) (61232017, 61102049, and 61161140454), partially supported by the Beijing Natural Science Foundation (4132053), and partially supported by the National Science and Technology Major Project (W13GY00040).
