R-learning-based team game model for Internet of things quality-of-service control scheme

Abstract

In modern times, it has been observed that Internet of things technology makes it possible for connecting various smart objects together through the Internet. For the effective Internet of things management, it is necessary to design and develop service models that ensure appropriate level of quality-of-service. Therefore, the design of quality-of-service management schemes has been a hot research issue. In this work, we formulate a new quality-of-service management scheme based on the IoT system power control algorithm. Using the emerging and largely unexplored concept of the R-learning algorithm and docitive paradigm, system agents can teach other agents how to adjust their power levels while reducing computation complexity and speeding up the learning process. Therefore, our proposed power control approach can provide the ability to practically respond to current Internet of things system conditions and suitable for real wireless communication operations. Finally, we validate the introduced concept and confirm the effectiveness of the proposed scheme in comparison with the existing schemes through extensive simulation analysis.

Introduction

The Internet of things (IoT) is regarded as a new technology and economic wave in the global information industry after the Internet.^1–3 In order to achieve anytime and anywhere functionality, the IoT equipment must connect, interact, and collaborate with the surrounding environment. For the effective IoT management, more quality of service (QoS) attributes must be considered, including information accuracy, coverage of IoT, required network resources, and energy consumption. Therefore, with regard to IoT services, QoS has been a popular research issue. Typically, different IoT applications have different QoS requirements. Nowadays, the requirement of QoS is considered to be an important non-functional element for the IoT system and must be guaranteed while implementing effective resource allocation and scheduling algorithms.^4,5

In current methods, in order to provide QoS to applications, real-time QoS schedulers are introduced in the IoT structure. ³ A QoS scheduler is an IoT system component designed to control the IoT system resources for various application services. In each local area, QoS schedulers assign the system resources to contending agents based on a set of criteria, namely, transmitter power, transmission rate, and QoS constraint. ⁵ During IoT system operations, these QoS schedulers aim to maximize system utilization while providing QoS requirements to classes of applications that have very tight requirements such as bit rate and delay. However, it is challenging to practically balance between network availability and QoS ensuring.^3,5,6

Game theory is a powerful tool to study situations of conflict and cooperation and is concerned with finding the best actions for individual decision makers, that is, players in these situations and recognizing stable outcomes. ⁷ It has been extensively used in microeconomics field, and only during the last years, it has received attentions as an effective method to design and model a distributed QoS control problems in telecommunications. ⁷ In this article, we develop a new game model called team game (TG). In our TG model, QoS schedulers are assumed as game players. All game players organize a team, and actions of all players are coordinated to ensure team cooperation by considering a combination of individual payoff as a team payoff. The main concept of TG is to extend the well-known Markov decision problem (MDP) to the multi-player case.

Traditionally, game theory assumes that players have perfect information about the game, enabling them to calculate all of the possible consequences of each strategy selection. ⁷ However, in real-world situations, a player must make decisions based on less-than perfect information. If a player does not have total information about the game, then it follows that the player’s reasoning must be heuristic. Therefore, to maximize IoT performance, the way in which agents learn network situations and make the best decisions by predicting the influence of others’ possible decisions is an important research issue in the field of IoT networking.^8–10

In recent years, many learning algorithms have been developed in an attempt to maximize system performance in non-deterministic settings.^8,11,12 Generally, learning algorithms guarantee that collective behavior converges to a coarse equilibrium status. In order to make control decisions in real time, QoS schedulers should be able to learn from dynamic system environments and adapt to the current network condition. Therefore, QoS schedulers using learning techniques acquire information from the environment, build knowledge, and ultimately improve their performance.^3,8,10

In 1993, A Schwartz ¹³ introduced an average-reward reinforcement learning algorithm called R-learning algorithm. Like Q-learning, R-learning algorithm uses the action value representation. In addition, it also needs to learn the estimate of the average reward. Therefore, R-learning algorithm is performed by the two time-scale learning process. In contrast to the value-iteration-based learning algorithms, the decision-learning approach in the R-learning algorithm allows an agent to directly learn the stationary randomized policy and directly updates the probabilities of actions based on the utility feedback. ¹⁴

Recently, there has been increasing interest in research of various R-learning algorithms. However, in this field, many problems still remain open. Even though the R-learning algorithm has received strong attentions, designing a novel R-learning algorithm for real-world problems is still difficult. There are many complicated restrictions, which are often self-contradictory and variable with the dynamic real-world IoT environment.

Docitive paradigm is an emerging technology to overcome the current limitations of R-learning algorithm. ¹⁵ Not only based on the cooperation to learn but also on the process of knowledge transfer, this paradigm can significantly speed up the learning process and increase precision. The docitive paradigm can provide a timely solution based on knowledge sharing in a cooperative fashion with other players in the IoT system, which allows game players to develop new capacities for selecting appropriate actions.^15,16

In this article, we develop a new IoT system power control scheme to ensure QoS provisioning. In the proposed scheme, we focus on how to integrate our TG game model and R-learning algorithm to tackle the QoS control problem in IoT systems. To implement the self-adaptability and real-time effectiveness, we adopt the docitive paradigm. Therefore, game players try to select optimal strategies in a distributed manner while approximating a common objective. According to the iterative TG game model, the proposed scheme attempts to ensure that individual decisions of players result in jointly optimal decisions for the players’ team. The main contributions of our study are as follows: (1) we develop a novel power control algorithm for IoT system, (2) we integrate the game theory and R-learning algorithm to tackle the power decisions, (3) we adopt the distributed learning approach to implement the self-adaptability and real-time effectiveness, and (4) we adopt the docitive paradigm to provide a timely solution. The most important novelty of our proposed scheme is the responsiveness to current IoT system conditions depending on the exchange of information and expert knowledge from other players, the so-called docitive players.

Related work

Over the years, a lot of state-of-the-art research work on the QoS control problem has been conducted. The QoS-based device-to-device communication (QDDC) scheme in the study of Dai et al. ¹⁷ was a novel device-to-device communication algorithm to enhance the spectrum efficiency. This scheme exploited the trade-off in power allocation of device-to-device transmitters while maximizing the number of allowed accessing device-to-device pairs under the QoS constraints. Then, the released communication resource would be separated into different device-to-device pairs. The QDDC scheme can be easily extended to the uplink case and the multiple channel case. ¹⁷

The deterministic sequencing of exploration and exploitation (DSEE) scheme ¹⁸ was developed as a new approach to the multi-armed bandit (MAB) problem, which is a class of sequential learning and decision problems with unknown models. Based on a DSEE, the DSEE scheme can find the minimum cardinality of the exploration sequence that ensures a reward loss in the exploitation sequence caused by incorrectly identified arm rank having an order no larger than the cardinality of the exploration sequence. ¹⁸

The non-Bayesian social learning (NBSL) scheme by Jiang et al. ¹⁹ studied how users in a dynamic system learned the uncertain system state and made multiple concurrent decisions by not only considering the current myopic utility but also taking into account the influence of subsequent users’ decisions. This scheme designed the recursive best response algorithms to find the subgame perfect Nash equilibrium for users and characterize special properties of the Nash equilibrium profile under homogeneous setting. ¹⁹

The multi-armed bandit with unknown dynamics (MBUD) scheme ²⁰ also considered the restless MAB problem with unknown dynamics in which a player chose one out of N arms to play at each time. The reward state of each arm transited according to an unknown Markovian rule when it was played and evolved according to an arbitrary unknown random process when it was passive. The MBUD scheme constructed a policy with an interleaving exploration and exploitation epoch structure that achieved a regret with logarithmic order. ²⁰ All the earlier work has attracted a lot of attention and introduced unique challenges to efficiently solve the QoS control problem. Compared to these schemes,^17,18 the proposed scheme attains better performance during the IoT system operations.

The rest of this article is organized as follows. First, the traditional MDP and R-learning algorithm are introduced in section “Markov decision process and R-learning algorithm.” Next, we formulate and explain the proposed TG model to solve the QoS problem in section “Proposed QoS control scheme for IoT systems.” In section “Performance evaluation,” we verify the effectiveness and efficiency of the proposed scheme from simulation results. Finally, we draw conclusions in section “Summary and conclusion.”

Markov decision process and R-learning algorithm

MDP is a mathematical framework for modeling decision making in situations and is useful for optimization problems solved via reinforcement learning. Based on inputs to a dynamic system, MDP probabilistically determines a successor state and continues for a finite or infinite number of stages.^7,8,12,21,22 Traditionally, MDP is defined as a tuple $\mathit{S},\mathbb{A},T,C(s(t),a(t)),\mathcal{T}$ , where $\mathit{S}=\{s_1,s_2,\dots ,s_n\}$ is the set of all possible states and $\mathbb{A}=\{a_1,a_2,\dots ,a_m\}$ is a finite set of admissible actions. $T=\{0,1,\dots ,t,t+1,\dots \}$ is a time, which is represented by a sequence of time steps. Let $s(t),a(t)$ be the state and action at time step t, respectively, where $s(t)\in \mathit{S}$ and $a(t)\in \mathit{S}$ . $C(s(t),a(t))$ is a cost at time t when the action $a(t)$ in state $s(t)$ occurs, and $C(s(t),a(t))\to \mathcal{R}$ . $\mathcal{T}$ is a state transition function $\mathcal{T}:\mathit{S}\times \mathbb{A}\to \Delta (\mathit{S})$ , where $\Delta (\mathit{S})$ is the set of discrete probability distributions over the set S . With the current state and action, the $\mathcal{T}$ function probabilistically specifies the next state of the environment.^3,8,21,22

The objective of the MDP is to find a policy that minimizes the cost of each state $s(t)$ . If $\Delta$ is the complete decision policy, the optimal cost value of state $s(t)$ can be written as

$$V^{*}\left(s\left(t\right)\right)=\underset{a\left(t\right)}{\mathbf{min}}[C(s\left(0\right),a\left(0\right))+\sum _{t=1}^{\infty }({\beta }^t\times C(s\left(t\right),a\left(t\right)))]$$

(1)

where $0\le {\beta }^t<1$ is a discount factor for the future state. Based on the principle of Bellman’s optimality, equation (1) can be re-written by a recursive equation as^3,21

$$V^{*}\left(s\right)=\underset{a}{\mathbf{min}}\left[C\left(s,a\right)+\beta \times {\displaystyle \sum }_{s^{\prime }\in S}\{P\left(s^{\prime }|s,a\right)\times V^{*}\left(s^{\prime }\right)\}\right],\mathrm{s}.\mathrm{t}.,a\in \mathbb{A}$$

(2)

where $s\prime$ represents all possible next states of S , and $P(s\prime |s,a)$ is the state transition probability from the state s to the state $s\prime$ . If we define $\Delta$ as the complete decision policy, we can specify the optimal policy as follows

$${\Delta }^{*}\left(s\right)=\underset{a}{\arg \mathbf{min}}[C(s,a)+\beta \times \sum _{s\prime \in \mathit{S}}\{P(s\prime |s,a)\times V^{*}(s\prime )\}]$$

(3)

To solve equation (3), reinforcement learning algorithms are common way. There exists several reinforcement learning algorithms. In this study, we adopt a novel average-reward reinforcement learning algorithm called R-learning. Like other reinforcement learning algorithms, R-learning algorithm uses the action value representation.^13,14 The action value ${\mathfrak{N}}^{\Delta }(s,a)$ represents the average adjusted value of doing an action a in state s once and then following policy $\Delta$ subsequently. ¹⁴ That is

$${\mathfrak{N}}^{\Delta }\left(s,a\right)=C\left(s,a\right)-{\eta }^{\Delta }+{\displaystyle \sum }_{s^{\prime }\in S}\left\{P\left(s^{\prime }|s,a\right)\times V^{*}\left(s^{\prime }\right)\right\}\mathrm{s}.\mathrm{t}.,V^{*}\left(s^{\prime }\right)=\underset{a\in \mathbb{A}}{\mathbf{min}}{\mathfrak{N}}^{\Delta }\left(s^{\prime },a\right)$$

(4)

where ${\eta }^{\Delta }$ is the average reward of policy $\Delta$ . In the R-leaning algorithm, system agents learn how to operate in the environment based on the effect of action and reward signal. In order to expediting the interaction between agent and environment efficiently, R-leaning algorithm is performed dynamically by a two-time-scale learning process.^13,14 However, conventional R-learning algorithm still has the limitation that the learning process of reinforcement learning system will be very slow.

Proposed QoS control scheme for IoT systems

In this section, we examine the applicability of MDP to design a novel power control algorithm and develop a new TG model for IoT systems. The proposed model can significantly improve the success rate of IoT services.

QoS-aware service management in the TG model

To develop our QoS-aware service management scheme, we assume and simplify the real-world situation for practical implementations:

1. Power strategies in each QoS schedulers are quantified. Usually, practicality is decided based on the computation complexity. Therefore, power levels should be simplified.

2. We assumed four QoS schedulers’ system in a local area like a conference room or small office part. Therefore, the proposed scheme has been developed as a four-player game model.

3. Pre-defined minimum bound $(\epsilon )$ is introduced for stable system status. It can be defined according to a subjective point of view for the system stability.

4. Heterogeneous traffic services are categorized into two classes according to the required QoS: class I (real-time) traffic services and class II (non real-time) traffic services. Class I data services are highly delay sensitive, and strict deadlines are applied. However, some flexible data services are called as class II traffic services; they are rather tolerant of delays.

5. If our model is applied in the situation with hundreds or thousands of QoS schedulers in a huge area, each QoS schedulers must be grouped and clustered in a distributed manner. Using a locally distributed approach, the proposed scheme is applied iteratively in each cluster.

In this study, we consider a new power control mechanism for IoT systems. Under the multi-QoS schedulers’ environment, we formulate the multiple decision-making process as a new TG model based on the multi-agent R-learning approach. Mathematically, the TG model $(\mathbb{T})$ can be defined as $\mathbb{T}=\{\mathit{S},N,{\mathit{A}}_{i,i\in N},T,U_{i,i\in N},\mathcal{T}\}$ at each time period $t\in T$ of gameplay:

• S is the set of all possible system states, which is the combinations of power levels of QoS schedulers.

• N is the set of all QoS schedulers.

• ${\mathit{A}}_i=\{{\mathcal{A}}_1^i,{\mathcal{A}}_2^i,\dots ,{\mathcal{A}}_m^i\}$ is the collection of power levels for the QoS scheduler $i\in N$ where m is the number of possible power levels. ${\mathcal{A}}_1^i$ and ${\mathcal{A}}_m^i$ are the pre-defined minimum and maximum power levels.

• $U_i$ is the payoff received by the QoS scheduler $i\in N$ .

• $\mathcal{T}$ is the state transition function and gives the probability to change the next state.

Usually, traditional solution concept of game theory is obtained with the following impractical assumptions: (1) fully rational players, (2) complete information, and (3) static game-model setting. These assumptions only hold in the theoretical and idealistic analysis. In the real-world IoT operations, it is impossible to transform the dynamic setting to static simulation setup. It means that the traditional solution with fully rational players is technically impossible to be obtained in the real world. To practically design the TG model, we develop a new solution concept called stable equilibrium (SE). Based on the R-learning and docitive paradigm, SE is applicable to the repeated choice in learning situations. For our TG, we assume that SE is a discrete set probability distributions over the available strategies chosen by all players.

Power control algorithm based on the R-learning algorithm

In the proposed power control scheme, we focus on how we can tackle the QoS control problem based on the R-learning algorithm approach. According to the self-adaptability, QoS schedulers in the proposed scheme can update the strategy based on the observations while responding to current IoT system conditions. Usually, the main interest of each QoS scheduler is to maximize the amount of transmitted data with the lower power consumption. However, there is a fundamental trade-off. To capture this conflicting relationship, a utility function $(U)$ is defined by considering the ratio of the throughput to transmit power. In our game model, the utility function for the ith QoS scheduler $(U_i)$ is defined as follows

$$U_i\left({\mathcal{A}}_j^i,{\mathbb{A}}_{-i}\right)=\frac{T_i\left(\mathbb{A}\right)}{{\mathcal{A}}_j^i},\mathrm{s}.\mathrm{t}.,{\mathcal{A}}_j^i\in {\mathbf{A}}_i,\mathbb{A}=\left\{{\mathrm{\Pi }}_{i\in N}{\mathcal{A}}_i|{\mathcal{A}}_i\in \left[{\mathcal{A}}_1^i,{\mathcal{A}}_m^i\right]\right\}$$

(5)

where ${\mathbb{A}}_{-i}$ is the transmit power vector without ${\mathcal{A}}_i$ , and $T_i(\mathbb{A})$ is the throughput of ith scheduler. Usually, throughput is defined as the number of information bits that are transmitted without error per unit-time. In wireless communications, it can be achieved with the signal-to-interference-plus-noise ratio (SINR) $({\gamma }_i)$ in the effective range. ⁷ Therefore, the throughput of the ith scheduler $(T_i(\mathbb{A}))$ can be expressed as

$$T_i\left(\mathbb{A}\right)=W\times {\log }_2(1+\frac{{\gamma }_i\left(\mathbb{A}\right)}{\Omega })$$

(6)

where W is the assigned channel bandwidth, and Ω (Ω ≥ 1) is the gap between uncoded M-ary quadrature amplitude modulation (M-QAM) and the capacity, minus the coding gain. ⁷ Finally, the ith scheduler’s utility is defined as follows

$$\underset{{\mathcal{A}}_j^i\in A_i}{\mathbf{max}}{U}_i=\underset{{\mathcal{A}}_j^i\in A_i}{\mathbf{max}}\left\{\frac{[W\times {\log }_2(1+\frac{{\gamma }_i\left(\mathbb{A}\right)}{\Omega })]}{{\mathcal{A}}_j^i}\right\}$$

(7)

In the developed game model, different schedulers can receive different payoffs for the same state transition. Based on our TG approach, schedulers seek to choose their power levels self-interestedly to maximize their payoffs. According to the R-learning equation (4), the expected payoff of QoS scheduler is

$${\mathfrak{N}}^{\Delta }\left(s,a\right)=U\left(s,a\right)-{\eta }^{\Delta }+{\displaystyle \sum _{s^{\prime }\in \mathbf{S}}\left\{P\left(s^{\prime }|s,a\right)\times \underset{a\in \mathbb{A}}{\mathbf{max}}{\mathfrak{N}}^{\Delta }\left(s^{\prime },a\right)\right\}}$$

(8)

In our TG game model, each QoS scheduler is interested in the goal of maximizing his utility function. In a distributed self-regarding fashion, each QoS scheduler in a dynamic IoT system learns the uncertain IoT situation and makes a power control decision by taking into account the online feedback mechanism. With an iterative repeating process, schedulers’ decision-making mechanism is developed based on the R-learning algorithm, which is an effective way for schedulers’ decision mechanism. Based on the dynamic learning mechanism, the developed algorithm can constantly adapt each QoS scheduler’s power level to get an appropriate performance balance between contradictory requirements.

Based on the feedback learning process, the proposed scheme can capture how schedulers adapt their power levels to achieve the better benefit. This procedure is defined as an online power control algorithm. In the proposed scheme, a selection probability for each power level strategy is dynamically changed based on the payoff ratio, which means the strategy convergence. Therefore, schedulers examine their payoffs periodically in an entirely distributed fashion. Without any impractical rationality assumptions, schedulers can modify their power levels in an effort to maximize their $\mathfrak{N}$ value in a distributed manner. Due to the online self-adjustment technique, this approach can significantly reduce the computational complexity and control overheads. Therefore, it is practical and suitable for the actual system implementation.

In equation (9), defining of $P(s\prime |s,a)$ is a probability decision problem. Through the multi-scheduler learning scheme, each scheduler adaptively learns the current IoT system situation to dynamically decide the $P(s\prime |s,a)$ . In the proposed scheme, each scheduler is assumed to be interconnected by letting them play in a TG with the same environment. Suppose there is a finite set of power levels ${\mathit{A}}_{i,1\le i\le N}(t)=\{{\mathcal{A}}_1^i(t)\dots {\mathcal{A}}_m^i(t)\}$ chosen by the scheduler i at the game iteration t. Correspondingly, the ${\mathit{U}}^i(t)=(u_1^i(t)\dots u_m^i(t))$ is a vector of specifying payoffs for the scheduler i. If the scheduler i plays the action ${\mathcal{A}}_{l,1\le l\le m}^i$ , it earns a payoff $u_{l,1\le l\le m}^i$ with probability $p_l^i$ . Over these actions, ${\mathit{P}}^i(t)=\{p_1^i(t),\dots ,p_m^i(t)\}$ is defined as the scheduler i’s probability distribution.

Power levels chosen by the schedulers are given as input to the environment, and the environmental response to these power levels serves as an input to each scheduler. Therefore, multiple schedulers are connected in a feedback loop with its environment. When a scheduler selects a power level with his respective probability distribution $\mathit{P}(·)$ , the environment produces a payoff $U(·)$ according to equation (5). Therefore, $\mathit{P}(·)$ should be adjusted adaptively in order to cope with the payoff fluctuation.

At every game round, all schedulers update their probability distributions based on the R-learning algorithm. If the scheduler i chooses ${\mathcal{A}}_j^i$ at the time t, this scheduler updates the jth action’s propensity $({\mathcal{J}}_j^i)$ as follows

$$\{\begin{array}{c}{\mathcal{J}}_j^i(t+1)={\mathcal{J}}_j^i\left(t\right)+{\xi }_j^i\left(t\right),\mathrm{s}.\mathrm{t}.,{\xi }_j^i\left(t\right)=\left(\frac{(u_j^i\left(t\right)-u_j^i(t-1))}{u_j^i(t-1)}\right)\hfill \\ \multicolumn{1}{c}{{\mathcal{J}}_k^i(t+1)={\mathcal{J}}_k^i\left(t\right),\mathrm{for}\mathrm{all}k\ne j}\end{array}$$

(9)

The QoS schedulers have to learn an effective action in a distributed fashion while achieving the common objective of IoT system. Known as multi-agent learning approach, we solve this problem using distributed R-learning algorithm and docitive paradigm. In the proposed TG model, the main challenge is how to ensure that individual decisions of each QoS scheduler approximate jointly optimal decisions for the team. As a docitive player, individual QoS schedulers cooperate with others by exchanging information while learning the action’s propensity from other team members, who are also performing power controls from the R-learning algorithm. In order to apply this approach, QoS schedulers periodically exchange their updated $\mathcal{J}$ values with team members. Based on the received $\mathcal{J}$ values, the jth action’s final propensity in the scheduler $i(\mathcal{F}\_{\mathcal{J}}_{j,1\le j\le m}^i(t+1))$ is given by

$$\begin{array}{l}ℱ\_{\mathcal{J}}_j^i\left(t+1\right)=\left(\sigma \times {\mathcal{J}}_j^i\left(t+1\right)\right)\\ +\left(\frac{\left(1-\sigma \right)}{\left|{\Gamma }_j\right|}\times {\displaystyle \sum _{k\in {\Gamma }_j}{\mathcal{J}}_j^k}\left(t+1\right)\right)\end{array}$$

(10)

where $\sigma$ is a weighted control parameter and ${\Gamma }_j$ is the set of schedulers, who send the updated $\mathcal{J}$ values about the jth action for the time $t+1$ . Finally, we can get the $P(s\prime |s,a)$ value in equation (8). The ith scheduler’s $P(·)$ value for the time $t+1(P^i(S\prime |S,{\mathcal{A}}_j^i(t+1)))$ is defined based on the proportion to each power level’s propensity

$$P^i(S\prime |S,{\mathcal{A}}_j^i(t+1))=\frac{\mathcal{F}\_{\mathcal{J}}_j^i(t+1)}{\sum _{l=1}^m\mathcal{F}\_{\mathcal{J}}_l^i(t+1)}$$

(11)

The main steps of proposed scheme

In this work, we discuss a new perspective in IoT systems to design the QoS control algorithm. In the proposed scheme, QoS schedulers adaptively decide their power levels while satisfying the QoS needs in coverage areas. Based on past actions and environmental feedback, we consider the R-learning algorithm and docitive paradigm that attempt to find out optimal actions effectively. Until now, several game models have been developed to help game players to learn from the dynamic network environment. An important feature in our TG model is to enable game players to reach quickly a certain desired game outcome.

From the result of individual learning experiences, each scheduler can learn how to effectively play under the dynamic network situations. Therefore, the payoff estimation at each game iteration can be used to update the $P(·)$ in such a way that those power levels with large payoff are more likely to be chosen again in the next iteration. To maximize their expected payoffs, QoS schedulers adaptively change their current power levels. This adjustment process is sequentially repeated until the change of probability $(P(·))$ is within a pre-defined minimum bound $(\epsilon )$ . The main steps of the proposed scheme can be described as follows:

• Step 1. At the initial time, $\mathcal{J}(·)$ is set to be equally distributed $(\mathcal{J}(·)=1/m,\mathrm{where}m\mathrm{is\; the\; number\; of\; power\; levels})$ , and control parameters $\sigma$ , N, W, Ω, $\alpha$ , $\beta$ , and $\epsilon$ are given to each QoS scheduler from the simulation scenario (refer to Table 1).

• Step 2. At the end of each game’s iteration, each QoS scheduler estimates independently its own payoff $(U(·))$ using equation (5).

• Step 3. Based on the currently received information, each QoS scheduler periodically adjusts $\mathcal{J}(·)$ values using equation (9).

• Step 4. According to the docitive paradigm, each QoS scheduler receives the $\mathcal{J}(·)$ values from other QoS schedulers. Using equation (10), the final propensity for each action is dynamically estimated.

• Step 5. Using the proportion to each strategy’s propensity, each $P(S\prime |S,\mathcal{A}(·))$ is defined according to equation (11).

• Step 6. Iteratively, each QoS scheduler selects a strategy $(\mathcal{A})$ to maximize his long-term expected payoff $(\mathfrak{N}(·))$ .

• Step 7. The sequential R-learning process is repeatedly operated in a distributed manner

  $$\{\begin{array}{c}{\mathfrak{N}}_{t+1}(s,a)\leftarrow {\mathfrak{N}}_t(s,a)\times (1-\beta )+\beta \times (r_{\Theta }(s,s\prime )-{\eta }_t+\underset{a\prime \in A}{\mathbf{max}}{\mathfrak{N}}_t(s\prime ,a\prime ))\hfill \\ \multicolumn{1}{c}{={\mathfrak{N}}_t(s,a)+\beta \times (r_{\Theta }(s,s\prime )-{\eta }_t+\underset{a\prime \in A}{\mathbf{max}}{\mathfrak{N}}_t(s\prime ,a\prime )-{\mathfrak{N}}_t(s,a))}\\ \multicolumn{1}{c}{{\eta }_{t+1}\leftarrow {\eta }_t\times (1-\alpha )+\alpha [r_{\Theta }(s,a)+\underset{a\prime \in A}{\mathbf{max}}{\mathfrak{N}}_t(s\prime ,a)-\underset{a\in A}{\mathbf{max}}{\mathfrak{N}}_t(s,a)]}\end{array}$$

  (12)

  where $r_{\Theta }(s,s\prime )$ is the $U(·)$ from the state s to the state $s\prime$ .

• Step 8. If all QoS schedulers reach the SE status, the game process is temporarily stopped. The SE status is formally defined as follows

  $$\begin{array}{c}\mathrm{SE}=\{\mathbb{P}={\left\{P^i(S\prime |S,{\mathcal{A}}_j^i\left(t\right))\right\}}_{i\in N,{\mathcal{A}}_j^i\in A_i}|\mathcal{M}\}\hfill \\ \mathrm{s}.\mathrm{t}.,\mathcal{M}=\underset{i,{\mathcal{A}}_j^i}{\mathbf{max}}(P^i(S\prime |S,{\mathcal{A}}_j^i\left(t\right))-P^i(S\prime |S,{\mathcal{A}}_j^i(t-1)))\le \epsilon \hfill \end{array}$$

  (13)

• Step 9. Constantly, each QoS scheduler is self-monitoring the current IoT situation. If the current system status is not the SE, it proceeds to Step 2 for the next iteration.

Table 1.

System parameters used in the simulation experiments.

Traffic class	Message application	Bandwidth requirement (Kbps)	Connection duration average
I	Delay-critical applications	32	30 s (0.5 min)
	Event-related applications	32	120 s (2 min)
		64	180 s (3 min)
II	General applications	128	120 s (2 min)
		256	180 s (3 min)
	Multimedia applications	384	300 s (5 min)
		512	120 s (2 min)
Parameter	Value	Description
$\sigma$	0.7	A weighted control parameter for the final propensity
|N|	4	The number of QoS schedulers
W	256 Kbps	Allocated bandwidth for service
Ω	1	Gap between uncoded M-QAM and the capacity, minus the coding gain
$\epsilon$	0.2	Pre-defined minimum bound for stable status
m	3	The number of power levels for QoS schedulers
$\beta$	0.3	A weighted control parameter for the R-learning algorithm
$\alpha$	0.3	A weighted control parameter for the R-learning algorithm

QoS: quality of service; M-QAM: M-ary quadrature amplitude modulation.

Performance evaluation

In this section, we compare the performance of the proposed scheme with other existing schemes. As mentioned in section “Introduction,” we select the QDDC scheme ¹⁷ and the DSEE scheme ¹⁸ and can confirm the performance superiority of our approach through the simulation analysis. The QDDC and DSEE schemes have been recently published and introduced unique challenges to efficiently solve the system control problems. The assumptions of our simulation environment are as follows:

• The simulated system consists of four QoS schedulers for a IoT system.

• In each scheduler covering area, a new service request is Poisson with rate $\rho$ (services/s), and the range of offered service load was varied from 0 to 3.0.

• The number of power levels (m) for QoS schedulers is three, and each strategy $({\mathcal{A}}_{j,1\le j\le m})$ is ${\mathcal{A}}_j\in \{50\mathrm{mW},75\mathrm{mW},100\mathrm{mW}\}$ .

• System performance measures obtained on the basis of 100 simulation runs are plotted as a function of the offered traffic load.

• The message size of each application is exponentially distributed with different means for different message applications.

• For simplicity, we assume the absence of physical obstacles in the experiments.

To facilitate the development and implementation of our simulator, Table 1 lists the system parameters.

In this section, the performance of the proposed scheme is compared with two existing schemes: the QDDC scheme ¹⁷ and the DSEE scheme. ¹⁸ Even though these existing schemes are recently published novel protocols, there are several disadvantages. First, these existing schemes rely on the impractical assumption for real operations; inapplicable presumption can cause potential erroneous decisions. Second, they cause the extra control overhead. The increased overhead can exhaust the system resources and need intractable computation. Third, these schemes cannot adaptively estimate the current system conditions. Fourth, these schemes operate the system by some fixed system parameters. Under dynamic real-world environments, it is an inappropriate approach to operate the IoT system.

Performance measures obtained through the simulation are IoT system throughput, service success probability, normalized service delay, system power stability, and application incomplete ratio. In Figures 1–5, the x-axis (a horizontal line) marks the service load intensities, which is varied from 0 to 3.0. Based on each rate of offered service load, performance criteria are evaluated as a normalized value; y-axis (a vertical line) represents the normalized value for each performance criteria.

Figure 1.

IoT system throughput.

Figure 2.

IoT service success probability.

Figure 3.

Normalized service delay in IoT systems.

Figure 4.

System power stability.

Figure 5.

Application incomplete ratio in IoT system.

Figure 1 shows the performance comparison of each scheme in terms of IoT system throughput. In this work, the IoT system throughput is measured as the normalized number of information bits that are transmitted without error per unit-time. Traditionally, it is one of the most critical aspects of the IoT management. The proposed TG-based approach adaptively decides power levels in an interactive-cooperative manner while monitoring the current system conditions. Therefore, the system throughput of the proposed scheme is better than the other schemes.

Figure 2 represents the service success probability of each IoT control scheme. In this work, service success probability is defined as the success ratio of service requests. In general, the excellent service success rate is a highly desirable property for actual IoT operations. As the offered traffic load increases, excessive service requests may lead to system congestion. Therefore, the service success probability decreases. This is intuitively correct. Under various application service requests, our game-based R-learning approach effectively handles the power control problem in IoT systems and leads to a better service success probability than other existing scheme.

Figure 3 reveals the normalized service delay in IoT systems. Usually, service delay is an important QoS metric and is able to reveal the fitness or unfitness of system protocols for different delay-sensitive applications. Due to the feedback-based repeated game approach, our proposed scheme can dynamically adapt the current situation and has much better accuracy than other existing schemes.

Figure 4 indicates the IoT system power stability of each scheme. In this study, the system power stability meant that the ratio of the actual power changes to the total power control periods. All the schemes have similar trends. However, our docitive paradigm–based power control policy can make the IoT network system more stable. Therefore, the proposed scheme can maintain the steady state under various network load intensities.

The curves in Figure 5 present the application incomplete ratio in the IoT System. As the offered traffic load increases, the IoT system will run out of the capacity for application service operations. Therefore, requested applications are likely to fail to meet the minimum QoS provisioning. Therefore, the application incomplete ratio increases linearly with the traffic load. From low to high traffic load intensities, the proposed scheme achieves a lower application incomplete ratio than other schemes.

The performance trends presented in Figures 1–5 are very similar. However, using the TG-based R-learning mechanism, the proposed scheme is flexible, adaptable, and able to sense the dynamic changing IoT system environment; it is essential in order to be close to the optimized system performance. Under diversified IoT traffic condition changes, the simulation result of proposed scheme is much better than the other schemes. Especially, the IoT system throughput, service success probability, normalized service delay, system power stability, and application incomplete ratio are improved by about 5%, 5%, 10%, 20%, and 10%, respectively, than the existing QDDC and DSEE schemes.^17,18

Summary and conclusion

The IoT is emerging as one of the major trends shaping the development of technologies in the Internet paradigm. The IoT technology has been evolving due to a convergence of multiple technologies, ranging from wireless communication to the Internet and from embedded systems to micro-electromechanical systems. However, the diversity of applications causes the QoS control problem in IoT platform. This study provides a novel QoS control algorithm for IoT systems. Based on the R-learning algorithm and docitive paradigm, we develop a new TG model. In the proposed game model, QoS schedulers iteratively observe the current IoT system conditions and adaptively change their power levels to maximize the system performance. Due to the self-regarding feature, these control decisions are made by an entirely distributed fashion. In actual IoT system operations, the distributed learning approach is suitable for ultimate practical implementation. Compared with the existing schemes, the simulation results showed that our proposed scheme effectively handles the current IoT system to achieve the better benefit.