Sage Journals: Discover world-class research

Abstract

Entropy is originally introduced to explain the inclination of intensity of heat, pressure, and density to gradually disappear over time. Based on the concept of entropy, the Second Law of Thermodynamics, which states that the entropy of an isolated system is likely to increase until it attains its equilibrium state, is developed. More recently, the implication of entropy has been extended beyond the field of thermodynamics, and entropy has been applied in many subjects with probabilistic nature. The concept of entropy is applicable and useful in characterizing the behavior of stochastic processes since it represents the uncertainty, ambiguity, and disorder of the processes without being restricted to the forms of the theoretical probability distributions. In order to measure and quantify the entropy, the existing probability of every event in the stochastic process must be determined. Different entropy measures have been studied and presented including Shannon entropy, Renyi entropy, Tsallis entropy, Sample entropy, Permutation entropy, Approximate entropy, and Transfer entropy. This review surveys the general formulations of the uncertainty quantification based on entropy as well as their various applications. The results of the existing studies show that entropy measures are powerful predictors for stochastic processes with uncertainties. In addition, we examine the stochastic process of lithium-ion battery capacity data and attempt to determine the relation between the changes in battery capacity over different cycles and two entropy measures: Sample entropy and Approximate entropy.

Keywords

Entropy uncertainty probability distribution stochastic processes characterization

Introduction

A stochastic process can be simply defined as a collection of random variables indexed by time. The continuous and discrete time cases can be separated to define stochastic processes more precisely. A discrete time stochastic process $X_{dis} = {X_{n}, n = 0, 1, 2, \dots}$ is a countable collection of random variables indexed by non-negative integers, whereas a continuous time stochastic process $X_{con} = {X_{t}, 0 \leq t \leq \infty}$ is an uncountable collection of random variables indexed by non-negative real numbers.¹

Real-world applications of most scientific subjects including stochastic processes are fraught with numerous uncertainties. Hence, it is believed, by many, that the stochastic processes, which consist of complicated uncertainties, are not predictable.^1–3 The applications of concepts of physics and thermodynamics to explain scientific phenomena have been studied in the recent decade. The concept of entropy, which stems from thermodynamics, has advanced our understanding of the world.^3–5 Entropy is one of the concepts in physics that can be useful in rejecting the null hypothesis of unpredictability of stochastic processes.^6–8 In this regard, various metrics including Shannon entropy, Renyi entropy, Tsallis entropy, Approximate entropy, Sample entropy, Transfer entropy, and Permutation entropy have been presented. The concept of entropy is applicable and useful in characterizing the behavior of stochastic processes since it represents the uncertainty, ambiguity, and disorder of the process without causing any restrictions on the theoretical probability distribution. In order to quantify the entropy, an associated probability distribution is needed.⁹

The conditional heteroscedastic models such as Generalized Autoregressive Conditional Heteroskedasticity (GARCH) models have traditionally been studied for measuring the variabilities in the stochastic processes due to the inherent uncertainties. In this article, first, we present traditional models as well as their assumptions and limitations. Next, we review the literature on the relationship between entropy measures and uncertainty inherent in stochastic processes. This research area is characterized by numerous studies reflecting different entropy measures, uncertainties, methodologies, and applications. We discuss nearly 140 scholarly works on the subject and draw general conclusions from previous studies. We attempt to provide a cogent approach to the issue, identify the key findings from the literature, and present the concluding remarks as well as the future research directions. In addition, we present case study of the lithium-ion battery capacity data, which is an example of stochastic process. We examine the relation between battery capacities over different cycles and two entropy measures: Sample entropy (SampEn) and Approximate entropy (ApEn).

Conditional heteroscedastic models

Before the idea of utilizing entropy measures for uncertainty modeling and quantification was introduced, long memory and volatility clustering^10,11 were traditionally studied for predictive modeling and uncertainty quantification based on conditional heteroscedastic models. Conditional heteroscedastic models are statistical models, which are usually used to model time series. These models explain the variance of the current error term, known as innovation, as a function of the previous periods’ error terms.¹² Examples of conditional heteroscedastic models are Autoregressive Conditional Heteroskedasticity (ARCH) model, Generalized ARCH (GARCH) model, GARCH in-Mean (GARCH-M) model, and Exponential GARCH (EGARCH) model. Conditional heteroscedastic models are mostly used to model the stochastic behavior of a time series, which is a special case of stochastic process with discrete time data. This section reviews and summarizes these models focusing on the main assumptions and modeling principles. Conditional heteroscedastic models for variances are based on Autoregressive (AR) and Moving Averages (MA) as follows.

The general form of the GARCH variance model, GARCH $(p, q)$ ,¹³ can be written as follows

\begin{matrix} Var (Y_{t} | Y_{t - 1}, \dots, Y_{t - p}) = σ_{t}^{2} \\ = ω + α_{1} Y_{t - 1}^{2} + \dots + α_{p} Y_{t - p}^{2} \\ + β_{1} σ_{t - 1}^{2} + \dots + β_{q} σ_{t - q}^{2} \\ = ω + \sum_{i = 1}^{p} α_{i} Y_{t - i}^{2} + \sum_{j = 1}^{q} β_{j} σ_{t - j}^{2} \end{matrix}

(1)

where $ω$ , $α_{i}$ , and $β_{j}$ are model parameters with the following assumptions

ω > 0, α_{i} \geq 0, β_{j} \geq 0, \sum_{i = 1}^{p} α_{i} + \sum_{j = 1}^{q} β_{j} < 1

(2)

It should be noted that large values of $Y_{t - 1}^{2}$ result in a large value of variance at time $t$ , which indicate that $Y_{t}$ is less predictable than the case with smaller values of $Y_{t - 1}^{2}$ . In other words, large values of $Y_{t - 1}^{2}$ indicate that $Y_{t}$ has more volatility.

In the GARCH-M model,¹⁴ the conditional variance, $h_{t}$ , of the random variable $X_{t}$ , given the random variable $Y_{t}$ , is a linear function of the previous periods’ conditional variances and the square of the past periods’ errors

X_{t} = \emptyset Y_{t} + γ h_{t} + ε_{t}

(3)

h_{t} = α_{0} + \sum_{i = 1}^{p} α_{i} ε_{t - i}^{2} + \sum_{i = 1}^{q} β_{i} h_{t - i}

(4)

In addition, the error terms, $ε_{t}$ , are distributed according to a normal distribution as follows

ε_{t} | Ω_{t - 1} ~ N (0, h_{t})

(5)

where $X_{t}$ is the risk premium, $Y_{t}$ is a predetermined vector of variables, $Ω$ is information set which explains the variability in the stochastic process $X_{t}$ , and $Ω_{t - 1}$ indicates that the current information is significant in regard to forecasting the future random variable $X_{t}$ . It should be noted that if information arrives in clusters, the random variable $X_{t}$ may exhibit clustering as well.

In the EGARCH model,¹⁵ the logarithm of the conditional variance follows an AR as shown below

X_{t} = a + γ e^{h_{t}} + b ε_{t - 1} + ε_{t}

(6)

\begin{matrix} \ln (h_{t}) & = ω + \sum_{i = 1}^{p} β_{i} \ln (h_{t - i}) \\ + \sum_{i = 1}^{q} α_{i} [\emptyset z_{t - i} + ψ (| z_{t - i} | - E | z_{t - i} |)] \end{matrix}

(7)

E | z_{t} | = {(\frac{2}{π})}^{1 / 2}

(8)

ε_{t} = e^{{(1 / 2)}^{h_{t}}} z_{t}

(9)

z_{t} ~ N (0, 1)

(10)

where $ω$ , $α_{i}$ , $β_{i}$ , $\emptyset$ , and $ψ$ are parameters. It is noted that $ω$ , $α_{i}$ , and $β_{i}$ are not necessarily non-negative. As it is shown in equations (6)–(10), the variances depend on both the sign and the size of lagged residuals.

In summary, GARCH family models can be used for modeling the stochastic processes with complicated endogenous and exogenous uncertainties such as the financial time series. In order to build the appropriate GARCH model that fits the data set, the GARCH parameters must be estimated based on the historical data. It is noted that the model order parameters $p$ and $q$ can be specified according to the autocorrelations of the standardized residuals. Hurst exponent is another metric which can be used for measuring the long-term memory of time series. Hurst exponent can be estimated by fitting the power law to the data set in time-series form. It represents the autocorrelations¹⁶ of the time series and how the autocorrelations decrease as the number of lags increases. Autocorrelation between the observations challenges the unpredictability of the stochastic processes since the future data can be forecasted based on the previous data. However, Gunaratne et al.¹⁷ stated that Hurst parameter, in isolation, is not sufficient to conclude the presence of long-term memory. Kwon and Yang¹⁸ showed that directionality between the different data points is not reflected in the cross-correlation.^19–21 Another approach for studying the phenomena of long memory and volatility clustering is by using measures based on the concept of entropy including Shannon entropy, Renyi entropy,^22,23 Tsallis entropy,^24–27 Transfer entropy, and Sample entropy as explained in the following section.

Entropy measures for uncertainty quantification

Shannon entropy

Entropy was first discussed by Clausius in 1865 to explore the inclination of intensity of heat, pressure, and density to gradually disappear over time. Based on this concept, Clausius established the Second Law of Thermodynamics stating that the entropy of an isolated system is likely to increase until it attains its equilibrium state. Schwill²⁸ and Shannon^29,30 extended the concept of entropy and claimed that entropy is not limited to thermodynamics and can be applied in any subject with probabilistic nature.^31,32 Bentes et al.³³ argued that physical concept of entropy is actually a special case of the Shannon entropy as it quantifies probabilities in the full state space.

Schwill²⁸ stated that entropy is a measure of uncertainty in random variables. For a random variable $X$ over a probability space $Ω$ and probability distribution $p (x)$ , Shannon entropy is defined as follows

H {(X)}_{cont} = - \int_{Ω} p (x) \cdot \log_{2} (p (x)) dx

(11)

H {(X)}_{discr} = - \sum_{x \in Ω} p (x) \cdot \log_{2} (p (x))

(12)

where $H (X)_{cont}$ and $H (X)_{discr}$ are entropy measures for continuous and discrete random variables, respectively. Moreover, equations (11) and (12) can be explained as follows

H (X) = E_{p} \log \frac{1}{p (X)}

(13)

As it is shown in equation (13), the expected value of $\log 1 / p (X)$ is equivalent to Shannon entropy. It should be noted that the original measurement units for entropy are bits, which are units of information needed to store one symbol in a message. Entropy can also be used as one way of measuring the uncertainty of random processes. The state with $p (X) = 0$ can be justified by $\lim_{p \to 0} p \cdot \log_{2} (p) = 0$ . It is also noted that, as Kullback and Leibler³⁴ showed, the definition of Shannon entropy lacks an invariant measure and in the absolutely continuous case depends on the parametrizations.³⁵ The concept of Shannon entropy has been applied to different domains including the power systems,³⁶ physics,^29,30 finance,³ biology,⁴ physiology,¹ and cardiovascular time series.^37–41

Maasoumi and Racine⁴² studied the predictability of the time series by utilizing a new measure of dependence, which performs similar to the Shannon entropy as follows

S = \frac{1}{2} \int_{- \infty}^{\infty} \int_{- \infty}^{\infty} {(f^{1 / 2} - f_{1}^{1 / 2} f_{2}^{1 / 2})}^{2} dxdy

(14)

where $f = f (x, y)$ is the joint density and $f_{1} = f (x)$ and $f_{2} = f (y)$ are the marginal densities of the random variables $X$ and $Y$ .

The interesting properties of this measure of dependence are as follows:

It is applicable in both discrete and continuous cases.

It lies within the range from 0 to unity. It is normalized to 0 in the case of independency and the modulus of the measure is 1 in the case of measurable exact relationship between the random variables.

In the case of a bivariate normal distribution, this measure of dependence has a simple relationship with the correlation coefficient.

It measures not only the distance but also the divergence.

Moreover, they compare their numerical results with those of a set of traditional measures. They show that their entropy measure^43–45 is capable of uncovering non-linear dependence in the time series. To the best of our knowledge, the work by Maasoumi and Racine is one of the first studies related to the entropy measures, uncertainty, and predictability of time series, at least, in the finance area.

Renyi and Tsallis entropies

Shannon entropy performs well only if the storage capacity of a transmitting channel is finite. Renyi presents a new entropy measure known as Renyi entropy of order $α$ , $S_{α} (X)$ , for discrete variables,^46,47 which is defined as follows

S_{α} (X) = \frac{1}{1 - α} \ln (\sum_{k = 1}^{n} p_{k}^{α})

(15)

where $α > 0$ and $α \neq 1$ . It is noted that in the case of limit $α \to 1$ , Renyi entropy and Shannon entropy are identical.

Tsallis entropy^24–27 for any non-negative real number $q$ , $S_{q} (X)$ , can be estimated as it is shown in the following formula

S_{q} (X) = \frac{1 - \sum_{i = 1}^{n} p_{i}^{α}}{q - 1}

(16)

Tsallis entropy yields power-law distribution, whereas Shannon entropy yields exponential equilibrium distribution. It is noted that $p_{k}^{α}$ and $p_{i}^{α}$ in equations (15) and (16) are the probability distributions of a given random variable $X$ . Renner and Wolf⁴⁸ introduced a new entropy measure called smooth Renyi entropy for characterizing the fundamental properties of a random variable such as the uniform randomness that may be extracted from the random variable. Let $B^{ε} (P) : = {Q : δ (P, Q) \leq ε}$ be the set of probability distributions which are ε-close to $P$ , the ε-smooth Renyi entropy of order $α$ can be defined as

H_{α}^{ε} (P) = \frac{1}{1 - α} \inf_{Q \in B^{ε} (P)} \log_{2} (\sum_{z \in Z} Q {(z)}^{α})

(17)

where $P$ is the probability distribution with range $Z$ , $δ$ is the variational distance, $α \in [0, \infty]$ , and $ε \geq 0$ .

Sample entropy

Sample entropy is useful in quantifying the fluctuation degree of a time series.^49–56 The sample entropy SampEn $(m, r, N_{c})$ can be defined as the negative natural logarithm of an estimate of the conditional probability that windows of length $m$ (subseries of the time series of length $N_{c}$ ) that remain similar within a tolerance $r$ also match at the next point.

The process of sample entropy measurement can be summarized as follows.

For a data sequence $x (n) = x (1), x (2), x (3), \dots, x (N)$ , where $N$ is the total number of data points to calculate SampEn $(r, m, N)$ , a run length $m$ and a tolerance window $r$ must be specified to compute SampEn $(r, m, N)$ .

Step 1

Form m-vectors, $X (1)$ to $X (N - m + 1)$ defined by

X (i) = [x (i), x (i + 1), \dots, x (i + m - 1)] i = 1 to N - m + 1

(18)

X (j) = [x (j), x (j + 1), \dots, x (j + m - 1)] j = 1 to N - m + 1

(19)

Step 2

Define the distance $d [X (i), X (j)]$ between vectors $X (i)$ and $X (j)$ as the maximum absolute difference between their respective scalar components

d [X (i), X (j)] = \max [| x (i + k) - x (j + k) |] k = 0 to m - 1

(20)

The relation matrix is shown in Figure 1.

Figure 1.

Relation matrix used for defining distance $d [X (i), X (j)]$ between vectors $X (i)$ and $X (j)$ .

Step 3

Define for each $i$

V^{m} (i) = no . of d_{m} [X (i), X (j)] \leq r i \neq j

(21)

B_{i}^{m} (i) = \frac{1}{N - m + 1} * V^{m} (i)

(22)

B^{m} (r) = \frac{1}{N - m} \sum_{i = 1}^{N - m} B_{i}^{m} (i)

(23)

Similarly

V^{m + 1} (i) = no . of d_{m + 1} [X (i), X (j)] \leq r i \neq j

(24)

A_{i}^{m} (i) = \frac{1}{N - m + 1} * V^{m + 1} (i)

(25)

A^{m} (r) = \frac{1}{N - m} \sum_{i = 1}^{N - m} A_{i}^{m} (i)

(26)

where $i = 1$ to $N - m + 1$ .

Step 4

Sample entropy for a finite data length of $N$ can be estimated as

SampEn (r, m, N) = - \ln [\frac{A^{m} (r)}{B^{m} (r)}]

(27)

Mutual information and transfer entropy

Entropy measures, including the measures discussed in the previous section, are useful in explaining the variability in univariate time series.^57–60 The concept of entropy is extended to the Transfer entropy⁶¹ and Mutual information,^62,63 which are discussed in the following section. The concept of Transfer entropy is applicable to uncovering the information flows between the systems, bivariate analysis of time series under uncertainties such as financial time series,²⁸ and multivariate analysis of time series in various disciplines such as physiology,⁶⁴ neuroscience,⁶⁵ ecology, bionomics, and neurology.²⁸

Mutual information and Transfer entropy measure the information flow between two time series and determine whether the variability in one random variable is helpful in explaining the variability in a second variable, whereas Shannon entropy is utilized for quantifying the variability in an individual random variable. Mutual information and Transfer entropy can be determined based on the conditional probability distributions between the two random variables and are discussed in this section.

The joint entropy $H (X, Y)$ and conditional entropy $H (X | Y)$ of two variables $X$ and $Y$ with probability space $X$ and $Y$ are defined as

H (X, Y) = - \sum_{x \in X, y \in Y} p (x, y) \cdot \log_{2} (p (x, y))

(28)

H (X | Y) = - \sum_{x \in X, y \in Y} p (x | y) \cdot \log_{2} (p (x | y))

(29)

Moreover, the relationship between conditional and joined entropies is as follows

H (X, Y) = H (X) + H (Y | X) = H (Y) + H (X | Y)

(30)

Mutual information of two random variables $X$ and $Y$ , $I (X; Y)$ ,^66,67 and its general form, Transfer entropy, determine the dependency between two time series.^68,69 Mutual information quantifies the reduction of uncertainty regarding $X$ by observing $Y$ ^70,71 as is shown in the following formula

\begin{matrix} I (X; Y) & = - \sum_{x \in X, y \in Y} p (x, y) \cdot \log_{2} \frac{p (x, y)}{p (x) p (y)} \\ = H (X) - H (X | Y) \end{matrix}

(31)

Figure 2 demonstrates the relationship between (1) different entropy measures as well as the joint entropy between the two random variables $X$ and $Y$ , (2) the entropy of $X$ conditioned on $Y$ , (3) the entropy of $Y$ conditioned on $X$ , and (4) mutual information between the random variables $X$ and $Y$ .

Figure 2.

The entropies $H (X)$ and $H (Y)$ , the joint entropy $H (X, Y)$ , the conditional entropies $H (X | Y)$ and $H (Y | X)$ , and mutual information $I (X; Y)$ .

Mutual information is symmetric and therefore cannot be used for determining the direction of an information flow^35,72

I (X; Y) = I (Y; X)

(32)

Mutual information can be modified in order to include lead-lag relationships⁷³ as follows

\begin{matrix} I {(X; Y)}_{τ} & = - \sum_{x_{n - τ} \in X, y_{n} \in Y} p (x_{n - τ}, y_{n}) \cdot \log_{2} \\ \frac{p (x_{n - τ}, y_{n})}{p (x_{n - τ}) p (y_{n})} \end{matrix}

(33)

where $τ$ is the number of lags. The mutual information between the random variables $X$ and $Y$ without lag and the mutual information between the random variable $X$ and the lagged version of the random variable $Y$ with $τ = - 1$ are depicted in Figure 3.

Figure 3.

The mutual information $I (X_{t}, Y_{t})$ and $I (X_{t}, Y_{t - 1})$ .

Measuring the interactions between the time series is critical in most disciplines. Correlation and cross-correlation can be utilized to detect the linear dependence between time series. However, these measures are not appropriate for modeling the dependence between the time series due to the following two reasons: time is not equally spaced and there is a non-linear mechanism in the time series. Hence, Transfer entropy, which performs according to transition probabilities, can be utilized to quantify the dependence between the time series.⁷⁴ Let $p (x_{1}, \dots, x_{n})$ be the probability of observing the sequence $(x_{1}, \dots, x_{n})$ . Transfer entropy,^75–78 which is a general form of mutual entropy, can be determined as

\begin{matrix} T_{Y \to X} (m, l) = \sum p (x_{t_{1}}, \dots, x_{t_{m}}, y_{t_{m - l + 1}}, \dots y_{t_{m}}) \cdot \\ \log_{2} \frac{p (x_{t_{m + 1}} | x_{t_{1}}, \dots, x_{t_{m}}, y_{t_{m - l + 1}}, \dots y_{t_{m}})}{p (x_{t_{m + 1}} | x_{t_{1}}, \dots, x_{t_{m}})} \end{matrix}

(34)

where $x_{t}$ and $y_{t}$ are the discrete states of $X$ and $Y$ at time $t$ , and the parameters $m$ and $l$ are the number of past observations included in $X$ and $Y$ . Figure 4 illustrates a bivariate analysis between the two random variables $X$ and $Y$ , and the Transfer entropy $T_{Y \to X} (1, 1)$ .

Figure 4.

The Transfer entropy $T_{Y \to X} (1, 1)$ .

The main difference between the Transfer entropy and Mutual information is that Transfer entropy takes into consideration the transition probabilities. Transfer entropy can be defined as the difference in the gain of information regarding $x_{t_{m + 1}}$ conditioned on both its own history and the history of $y_{t}$ , and the case where it is conditioned on its own history only. It is noted that Transfer entropy is not symmetric. Moreover, for a stationary Markov process, only the most recent history, $X_{n - 1}$ , is required for forecasting $X_{n}$ .

Marschinski and Kantz⁷⁹ examined the well-known physical concepts such as Spin systems,⁸⁰ Turbulence,⁸¹ Universality,⁸² Self-organized criticality,^83,84 Complexity,⁸⁵ and entropy and concluded that among these physical concepts entropy may be useful in capturing the uncertainties inherent in stochastic processes. They define the Transfer entropy as follows

\begin{matrix} Transfer Entropy (TE) = {information about future data I (t + 1) gained form past joint \\ observations of I and J} - {information about future data I (t + 1) gained from past \\ observations of I only} = information flow from J to I \end{matrix}

(35)

Kwon and Yang⁸⁶ argued that the first step in the study of Transfer entropy is to discretize the time series^87,88 by some coarse graining. They partition the variables $x_{n}$ into discretized counterparts $A_{n}$ , by utilizing a constant $d$ as follows

A_{n} = 0 for x_{n} \leq - \frac{d}{2} (decrease)

(36)

A_{n} = 1 for - \frac{d}{2} \leq x_{n} \leq \frac{d}{2} (intermediate)

(37)

A_{n} = 2 for x_{n} \geq \frac{d}{2} (increase)

(38)

It is crucial to determine an appropriate constant $d$ since the probability of the states varied by the changes in $d$ .

Effective transfer entropy

Using the longer history of the time series in the calculation of Transfer entropy may cause noises rather than providing more information.^89–92 The limitation of the Transfer entropy is that it does not account for the effect of noises.⁶⁷ In order to take into consideration the noises, Effective Transfer entropy⁹³ is introduced as

E T_{Y \to X} (m, l) = T_{Y \to X} (m, l) - μ ({\hat{T}}_{sh})

(39)

where ${\hat{T}}_{sh}$ is the mean shuffled Transfer entropy,²⁸ $m$ is the length of the past history of $X$ , and $l$ is the length of the past observations of $Y$ .

Relative explanation added (REA) can be utilized as a secondary measure²⁸ to study the uncertainty based on the concept of entropy as follows

REA (m, l) = \frac{H (X_{0} | X_{- 1}, \dots, X_{- m}, Y_{- 1}, \dots, Y_{- l})}{H (X_{0} | X_{- 1}, \dots, X_{- m})} - 1

(40)

REA quantifies the extra information obtained from the history of the random variables $X$ and $Y$ , when $X$ has already been observed.

Entropy rate

Entropy rate measures the average information required in order to predict a future observation given the previous²⁸ observations as follows

h_{\infty} = \lim_{n \to \infty} H (X_{0} | X_{- 1}, \dots, X_{- (n + 1)})

(41)

where $H (X_{0} | X_{- 1}, \dots, X_{- (n + 1)})$ represents the entropy measure conditioned on the history of the random variable $X$ . The conditional entropy $H (X_{0} | X_{- 1})$ and the entropy rate $h_{\infty}$ are illustrated in Figure 5.

Figure 5.

The conditional entropy $H (X_{0} | X_{- 1})$ and the entropy rate $h_{\infty}$ .

Normalized permutation entropy and number of forbidden patterns

Zunino et al.⁹⁴ stated that the Random Walk model is based on the assumption that the dynamic changes in the data are uncorrelated and therefore entropy measures can be used to capture the uncertainties inherent in the time series.⁹⁵ The authors discussed the Second Law of Thermodynamics, which states that entropy expands monotonically over time. In addition, they examined Renyi,^96,97 Tsallis,⁹⁸ Approximate,^99–101 and Transfer entropies, as well as a local approach to study Shannon entropy.¹⁰² Zunino et al.¹⁰³ presented two new quantifiers for predictive models: the number of Forbidden patterns and the normalized Permutation entropy. Both of these metrics are model independent and therefore have wider applicability. The number of Forbidden patterns is positively correlated with the inefficiency degree, and the normalized Permutation entropy is negatively correlated with the inefficiency degree. Forbidden patterns represent the patterns that cannot be uncovered because of the underlying deterministic structure of the time series. It is noted that Permutation entropy is the normalized form of Shannon entropy over the probability distribution $P = {p (π_{i}), i = 1, \dots, D!}$ as follows

H_{S} [P] = \frac{S [P]}{S_{\max}} = \frac{[- \sum_{i = 1}^{D!} p (π_{i}) \ln (p (π_{i}))]}{S_{\max}}

(42)

where $S$ stands for Shannon entropy and $S_{\max} = \ln D! (0 \leq H_{S} \leq 1)$ . It is noted that the greatest value of $H_{S} [P]$ is equal to unity in the case of totally random sequence and the smallest value of $H_{S} [P]$ is equal to zero in the case of increasing/decreasing sequence.

Entropy measures based on singular value decomposition

Gu et al.¹⁰⁴ investigated the predictive ability of the Singular Value Decomposition entropy. The multifractality of time series is due to local fluctuations of the index, long-term memory of the volatility,^105,106 heavy-tailed distributions, herding behavior, outside information, the intrinsic properties, non-linear incremental changes in the data, and so on. According to their study, the tendency of the entropy series is ahead of the component index, which indicates that the Singular Value Decomposition entropy has predictive power for the variable.

Caraiani¹⁰⁷ studied the entropy measures based on the Singular Value Decomposition of the correlation matrix for the components of the time series and defined the correlation matrix of data as follows

R_{i, j} = \frac{〈 (y_{i} - 〈 y_{i} 〉) - (y_{j} - 〈 y_{j} 〉) 〉}{σ_{i} σ_{j}}

(43)

where $〈 y_{i} 〉$ is the mean, and $σ_{i}$ and $σ_{j}$ are the standard deviations of $y_{i}$ and $y_{j}$ , respectively. He used Granger causality tests^108–110 to determine whether a certain time series is helpful in predicting the future dynamics of the second variable as is shown in the following formula

y_{t} = β_{0} + \sum_{k = 1}^{N} β_{k} y_{t - k} + \sum_{t = 1}^{N} α_{l} x_{t - l} + u_{t}

(44)

where $y_{t}$ and $x_{t}$ are the variables of interest and $u_{t}$ are the uncorrelated disturbances. The parameters $k$ and $l$ are the order of lags. The null and alternative hypotheses in his statistical analysis are as follows

H_{0} : α_{l} = 0 for any l

(45)

H_{1} : α_{l} \neq 0 for at least some l

(46)

Residual sum of squares for the initial series, $RS S_{u}$ , and the restricted version without $y_{t - k}$ , $RS S_{v}$ , can be defined as follows

RS S_{u} = \sum_{t = 1}^{N} u_{t}^{2}

(47)

RS S_{v} = \sum_{t = 1}^{N} v_{t}^{2}

(48)

The appropriate statistic, $F_{0}$ , can be determined as follows

F_{0} = \frac{(RS S_{u} - RS S_{v}) / l}{RS S_{u} / (N - 2 l - 1)}

(49)

F ₀-statistic has $l$ and $(N - l - 1)$ degrees of freedom. Caraiani¹⁰⁷ used the F₀-statistic and attempted to address the following question: do the entropy indices Granger-cause the time series? In other words, are the past values of a random variable, $x_{t}$ , helpful in predicting a second variable, $y_{t}$ , beyond the information contained in the past values of $y_{t}$ ?

Approximate entropy

In this section, other measures of dependencies of random variables are presented. Different methods for quantifying the uncertainties of stochastic processes have been introduced including the methods based on Fourier analysis,^111,112 Symbolic analysis,¹¹³ Amplitude statistics,^113–117 and Wavelet transform.^118,119 Huang et al.¹²⁰ reported that real-world complex dynamics reflect anomalous, chaotic, irregular, and non-stationary characterizations due to various factors, including internal drive and external disturbances. In particular, the methods of analyzing the stochastic process of financial times series include wavelet analysis,^121,122 detrended fluctuation analysis,^123,124 and diffusion entropy analysis (DEA).^125–127 Shi and Shang¹²⁸ studied two entropy-related metrics, Cross-sample entropy and Transfer entropy, as well as a Cross-correlation analysis to examine the relationship between time series among different data sets. Cross-sample entropy, Transfer entropy, and Cross-correlation measures are utilized to obtain the asynchrony, information flow, and cross-correlation between the time series, respectively. Other statistical tools, which have been introduced to examine the stochastic processes, are Correlation function, Multi-fractal, Spin-glass models, and Complex networks.^129–137 The Kolmogorov–Sinai entropy for real-world time series cannot be precisely estimated. Therefore, the analysis of the finite and noisy time series can be performed by using Approximate entropy.¹³⁷ In addition, Cross-sample entropy is based on Approximate entropy,^138–141 which captures changes in a system complexity. Approximate entropy can be utilized to measure patterns in time series and uncover the regularities by extracting the noises from the original data. Kristoufek and Vosvrda¹³⁵ studied various variables including the Efficiency indices, Long-term memory, Fractal dimension,¹⁴² and Approximate entropy. According to their study, the efficiency index can be defined as a distance from the unpredictable process specification based on various measures including Long-term memory, Fractal dimension, and Approximate entropy.

Consider a series of numbers of length $N$ , $u (1), \dots, u (N)$ and form m-blocks of the subseries $x (i) \equiv (u (i), u (i + 1) \dots, u (i + m - 1))$ , where $m$ is a non-negative number less than or equal to the length of the series. We then determine $C_{i}^{m} (r)$ as the fraction of blocks $x (j)$ which have a distance of less than a given positive real number $r$ . Approximate entropy can be calculated as¹⁴⁰

ApEn (m, r, N) (u) = Φ^{m} (r) - Φ^{m + 1} (r), m \geq 0

(50)

where $Φ^{m} (r)$ is defined as

Φ^{m} (r) = \frac{1}{N - m + 1} \sum_{i = 1}^{N - m + 1} {logC}_{i}^{m} (r)

(51)

and $C_{i}^{m} (r)$ can be calculated as

C_{i}^{m} (r) = \frac{1}{N - m + 1} \sum_{j = 1}^{N - m + 1} Θ (r - d (x (i), x (j)))

(52)

$Θ$ is Heaviside function which counts the instances where the distance $d$ is below the threshold $r$ . Approximate entropy quantifies the logarithmic likelihood that sequences of patterns that are close for $m$ observations remain close on next comparisons as well.²⁸

Case study

Battery health monitoring is important since the failure of the battery may result in the failure of the system. The proper and efficient battery health management can prevent the disastrous hazards and premature failures. This section aims at determining the correspondence between the loss in the capacity of the battery and the Sample entropy (SampEn) and Approximate entropy (ApEn) of voltages. Many approaches for estimating the battery capacity are based on a direct analysis of battery capacity with respect to aging cycle. Approximate entropy is capable of quantifying the regularity of time series and therefore can be used as estimators of degradation of the battery capacities.

In this section, we use the definitions of Sample entropy and Approximate entropy and determine these two entropy measures for the changes in the battery voltages over time. We examine five lithium-ion batteries as follows: PL19, PL17, PL11, PL10, and PL09. The battery voltages over time are illustrated in Figure 6. Figure 7 displays the changes in the battery capacities over cycle. The Sample entropy (SampEn) and Approximate entropy (ApEn) are calculated and shown in Figures 8 and 9, respectively.

Figure 6.

Battery voltages versus cycles.

Figure 7.

Battery capacities versus cycles.

Figure 8.

Sample entropy (SampEn) versus cycles.

Figure 9.

Approximate entropy (ApEn) versus cycles.

It is noted that the degradation of batteries may be due to the internal shorts, opening of the short, or cell undergoing reversal. Thus, detecting these causes of degradation is very significant for battery diagnosis in real-world applications.

Discussion and conclusion

Entropy was originally defined as the average number of bits needed to store or communicate one symbol in a message. It can also quantify the uncertainty involved in predicting the value of a random variable. In this article, we review various entropy measures to examine the uncertainties inherent in stochastic processes. It is shown that different predictive models have been built by using various entropy measures. The concept of entropy has not been limited to physics and thermodynamics. The entropy measures have been used for developing predictive models in various disciplines including mechanics, finance, physiology, neuroscience, ecology, bionomics, and neurology. Moreover, various entropy measures have been introduced depending on the process and time series studied. These metrics include, but are not limited to Shannon entropy, Approximate entropy, Sample entropy, Transfer entropy, and Permutation entropy. The concept of entropy is helpful in characterizing stochastic processes since it represents the uncertainty and disorder of the process, without causing any restrictions on the theoretical probability distribution. The contribution of this article lies in its description of information, uncertainty, entropy, and ignorance in stochastic processes. More precisely, information is the decline in disorder and ambiguity, uncertainty is referred to the unlikelihood of logical reasoning, entropy is the expected information, and ignorance is the lack of knowledge regarding the uncertainty. The key findings of this review and the future research directions are as follows:

Entropy is useful in explaining the uncertainties inherent in stochastic processes.

Entropy measures have predictive power, and the type of time series studied may have an impact on the predictive ability of the entropy measures.

Entropy measures play an important role in various branches of science and disciplines.

Entropy is an indicator representing the complexity of the signals. The greater the entropy measures are, the more complexity the signals accumulate, and the more information the signals contain.

The results of the existing studies in this research area show that the predictive ability of the entropy measures is related to the complex degree of the available information.

Future work should examine the predictive power of the entropy after taking into account the effect of other variables such as periodicity. Variables such as periodicity may make some of the uncertainties inherent in stochastic processes more explicit.

The entropy-based proposed method can be applied to battery monitoring and prognostics, and it is found that Sample entropy and Approximate entropy are useful in quantifying the fluctuation degree of a stochastic process. These two entropy measures can quantify the regularity of time series and assess the predictability of a data sequence. Thus, they can be applied to battery capacity data as an indicator for battery health.

Footnotes

Handling Editor: James Baldwin

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iDs

Alireza Namdari

Zhaojun (Steven) Li

References

Ross

Kelly

Sullivan

et al . Stochastic processes. New York: Wiley, 1996.

Bag

Banik

Ray

DS.

Noise properties of stochastic processes and entropy production. Phys Rev E 2001; 64: 026110.

Robinson

PM.

Consistent nonparametric entropy-based testing. Rev Econ Stud 1991; 58: 437–453.

Sabatini

AM.

Analysis of postural sway using entropy measures of signal complexity. Med Biol Eng Comput 2000; 38: 617–624.

Schürmann

Bias analysis in entropy estimation. J Phys A Math Gen 2004; 37: L295.

Billingsley

Ergodic theory and information. New York: Wiley, 1965.

Ebrahimi

Maasoumi

Soofi

Comparison of entropy and variance orderings. J Econometrics 1999; 90: 317–336.

Faddeev

DK.

On the concept of entropy of a finite probabilistic scheme. Uspekhi Mate Nauk 1956; 11: 227–231.

Zurek

WH.

Complexity, entropy and the physics of information. Boca Raton. FL: CRC Press, 2018.

10.

Kullmann

Kertesz

Mantegna

RN.

Identification of clusters of companies in stock indices via Potts super-paramagnetic transitions. Phys A Stat Mech Appl 2000; 287: 412–419.

11.

Nanda

Mahanty

Tiwari

MK.

Clustering Indian stock market data for portfolio management. Expert Syst Appl 2010; 37: 8793–8798.

12.

Engle

Focardi

Fabozzi

FJ.

ARCH/GARCH models in applied financial econometrics. Encyclopedia of Financial Models, 22 October 2012, http://pages.stern.nyu.edu/∼rengle/ARCHGARCH.pdf

13.

Katsiampa

Volatility estimation for Bitcoin: a comparison of GARCH models. Econ Lett 2017; 158: 3–6.

14.

Christensen

Dahl

Iglesias

EM.

Semiparametric inference in a GARCH-in-mean model. J Econometrics 2012; 167: 458–472.

15.

Hansen

Huang

Exponential GARCH modeling with realized measures of volatility. J Bus Econ Stat 2016; 34: 269–287.

16.

Mizuno

Takayasu

Correlation networks among currencies. Phys A Stat Mech Appl 2006; 364: 336–342.

17.

Gunaratne

McCauley

Bassler

KE.

Markov processes, Hurst exponents, and nonlinear diffusion equations: with application to finance. Phys A Stat Mech Appl 2006; 369: 343–353.

18.

Kwon

Yang

JS.

Information flow between stock indices. Europhys Lett 2008; 82: 68003.

19.

Podobnik

Stanley

HE.

Detrended cross-correlation analysis: a new method for analyzing two nonstationary time series. Phys Rev Lett 2008; 100: 084102.

20.

Podobnik

Jiang

Zhou

et al . Statistical tests for power-law cross-correlated processes. Phys Rev E 2011; 84: 066118.

21.

Shadkhoo

Jafari

GR.

Multifractal detrended cross-correlation analysis of temporal and spatial seismic data. Eur Phys J B 2009; 72: 679–683.

22.

Csiszár

Generalized cutoff rates and Rényi’s information measures. IEEE T Inform Theory 1995; 41: 26–34.

23.

Jizba

Kleinert

Shefaat

Rényi’s information transfer between financial time series. Phys A Stat Mech Appl 2012; 391: 2971–2989.

24.

Tsallis

Entropic nonextensivity: a possible measure of complexity. Chaos Soliton Fract 2002; 13: 371–391.

25.

Tsallis

Nonadditive entropy: the concept and its use. Eur Phys J A 2009; 40: 257–266.

26.

Tsallis

Possible generalization of Boltzmann-Gibbs statistics. J Stat Phys 1988; 52: 479–487.

27.

Tsallis

Introduction to nonextensive statistical mechanics: approaching a complex world. Berlin: Springer Science+Business Media, 2009.

28.

Schwill

Entropy analysis of financial time series. 2018, https://arxiv.org/pdf/1807.09423.pdf

29.

Shannon

CE.

Communication theory of secrecy systems. Bell Syst Tech J 1949; 28: 656–715.

30.

Shannon

CE.

A mathematical theory of communication. ACM SIGMOBILE Mobile Comput Commun Rev 2001; 5: 3–55.

31.

Efron

Tibshirani

RJ.

An introduction to the bootstrap. Boca Raton, FL: CRC Press, 1994.

32.

Fast

JD.

Entropy: the significance of the concept of entropy and its applications in science and technology. London: Macmillan, 1982.

33.

Bentes

Menezes

Mendes

DA.

Long memory and volatility clustering: is the empirical evidence consistent across stock markets?

Phys A: Stat Mech Appl 2008; 387: 3826–3830.

34.

Kullback

Leibler

RA.

On information and sufficiency. Ann Math Stat 1951; 122: 79–86.

35.

Jaynes

ET.

Information theory and statistical mechanics. Phys Rev 1957; 106: 620.

36.

Cai

Qian

. A study of wavelet entropy theory and its application in power system. In: Proceedings of the international conference on intelligent mechatronics and automation, Chengdu, China, 26–31 August 2004, pp.847–851. New York: IEEE.

37.

Dawes

Moulden

Sheil

et al . Approximate entropy, a statistic of regularity, applied to fetal heart rate data before and during labor. Obstet Gynecol 1992; 80: 763–768.

38.

Richman

Moorman

JR.

Physiological time-series analysis using approximate entropy and sample entropy. Am J Physiol Heart Circ Physiol 2000; 278: H2039–H2049.

39.

Fleisher

Pincus

Rosenbaum

SH.

Approximate entropy of heart rate as a correlate of postoperative ventricular dysfunction. Anesthesiology 1993; 78: 683–692.

40.

Palazzolo

Estafanous

Murray

PA.

Entropy measures of heart rate variation in conscious dogs. Am J Physiol Heart Circy Physiol 1998; 274: H1099–H1105.

41.

Pincus

Viscarello

RR.

Approximate entropy: a regularity measure for fetal heart rate analysis. Obstet Gynecol 1992; 79: 249–255.

42.

Maasoumi

Racine

Entropy and predictability of stock market returns. J Econometrics 2002; 107: 291–312.

43.

Yang

et al . Analysis of heart rate variability based on singular value decomposition entropy. J Shanghai Univ Eng Ed 2008; 12: 433.

44.

Lian

Zhang

JW.

Vibration of hydraulic structure modal identification method ERA based on order reduction of singular entropy. Sci China Ser E Sci Technol 2008; 38: 1398–1413.

45.

Ljung

Box

GE.

On a measure of lack of fit in time series models. Biometrika 1978; 65: 297–303.

46.

Jizba

Arimitsu

The world according to Rényi: thermodynamics of multifractal systems. Ann Phys 2004; 312: 17–59.

47.

Rényi

On measures of entropy and information. Budapest: Hungarian Academy of Sciences, 1961.

48.

Renner

Wolf

Smooth Rényi entropy and applications. In: Proceedings of the international symposium on information theory, Chicago, IL, 27 June–2 July 2004, p.233. New York: IEEE.

49.

Guo

Shi

A sampled entropy-based method for estimating lithium ion battery capacity under non-standard operation condition. In: Proceedings of the IIE annual conference, Pittsburgh, PA, 20–23 May 2017, pp.43–48. Norcross, GA: Institute of Industrial and Systems Engineers (IISE).

50.

Guo

A new feature extraction algorithm based on entropy cloud characteristics of communication signals. Math Prob Eng 2015; 2015: 1–8.

51.

Sun

Jou

JC.

Auxiliary diagnosis method for lead–acid battery health based on sample entropy. Energ Convers Manage 2009; 50: 2250–2256.

52.

Widodo

Shim

Caesarendra

et al . Intelligent prognostics for battery health monitoring based on sample entropy. Expert Syst Appl 2011; 38: 11763–11769.

53.

Zheng

Han

et al . Lithium ion battery pack power fade fault identification based on Shannon entropy in electric vehicles. J Power Sources 2013; 223: 136–146.

54.

Jia

et al . Enhanced sample entropy-based health management of Li-ion battery for electrified vehicles. Energy 2014; 64: 953–960.

55.

Pan

Chen

CP.

Intelligent prognostics for battery health monitoring using the mean entropy and relevance vector machine. IEEE T Syst Man Cy Syst 2014; 44: 851–862.

56.

Jiang

Cao

et al . Battery health prognosis for electric vehicles using sample entropy and sparse Bayesian predictive modeling. IEEE T Ind Electron 2016; 63: 2645–2656.

57.

Rethinam

Rajagopalan

Janakiraman

et al . Jitters through dual clocks: an effective entropy source for true random number generation. In: Proceedings of the international conference on computer communication and informatics (ICCCI), Coimbatore, India, 4–6 January 2018, pp.1–5. New York: IEEE.

58.

Tsur

The effective entropy of next/previous larger/smaller value queries. Inform Process Lett 2019; 145: 39–43.

59.

Costa

Goldberger

AL.

Generalized multiscale entropy analysis: application to quantifying the complex volatility of human heartbeat time series. Entropy 2015; 17: 1197–1203.

60.

Lin

et al . Analysis of complex time series using refined composite multiscale entropy. Phys Lett A 2014; 378: 1369–1374.

61.

James

Barnett

Crutchfield

JP.

Information flows? A critique of transfer entropies. Phys Rev Lett 2016; 116: 238701.

62.

Zheng

Pan

et al . Use of mutual information and transfer entropy to assess interaction between parasympathetic and sympathetic activities of nervous system from HRV. Entropy 2017; 19: 489.

63.

Villaverde

Ross

Morán

et al . MIDER: network inference with mutual information distance and entropy reduction. PLoS ONE 2014; 9: e96732.

64.

Montalto

Faes

Marinazzo

. MuTE: a new Matlab toolbox for estimating the multivariate transfer entropy in physiological variability series. In: Proceedings of the 8th conference of the European Study Group on cardiovascular oscillations, Trento, 25–28 May 2014, pp.59–60. New York: IEEE.

65.

Wibral

Vicente

Lindner

Transfer entropy in neuroscience. In: Wibral

Vicente

Lizier

(eds) Directed information measures in neuroscience. Berlin: Springer, 2014, pp.3–36.

66.

Blumentritt

Schmid

Mutual information as a measure of multivariate association: analytical properties and statistical estimation. J Stat Comput Simul 2012; 82: 1257–1274.

67.

Schreiber

Measuring information transfer. Phys Rev Lett 2000; 85: 461–464.

68.

Dimpfl

Peter

FJ.

Using transfer entropy to measure information flows between financial markets. Stud Nonlin Dynam Econ 2013; 17: 85–102.

69.

Pincus

Singer

BH.

Randomness and degrees of irregularity. Proc Natl Acad Sci U S A 1996; 93: 2083–2088.

70.

Dionisio

Menezes

Mendes

DA.

Mutual information: a measure of dependency for nonlinear time series. Phys A Stat Mech Appl 2004; 344: 326–329.

71.

Tumminello

Aste

Di Matteo

et al . A tool for filtering information in complex systems. Proc Natl Acad Sci U S A 2005; 102: 10421–10426.

72.

Golan

Information and entropy econometrics—a review and synthesis. Found Trends Econ 2008; 2: 1–45.

73.

Cover

Thomas

JA.

Elements of information theory. Hoboken, NJ: John Wiley & Sons, 2012.

74.

Geweke

Measurement of linear dependence and feedback between multiple time series. J Am Stat Assoc 1982; 77: 304–313.

75.

Vicente

Wibral

Lindner

et al . Transfer entropy–a model-free measure of effective connectivity for the neurosciences. J Comput Neurosci 2011; 30: 45–67.

76.

Staniek

Lehnertz

Symbolic transfer entropy. Phys Rev Lett 2008; 100: 158101.

77.

Bauer

Cox

Caveness

et al . Finding the direction of disturbance propagation in a chemical process using transfer entropy. IEEE T Control Syst Technol 2007; 15: 12–21.

78.

Ito

Hansen

Heiland

et al . Extending transfer entropy improves identification of effective connectivity in a spiking cortical network model. PLoS ONE 2011; 6: e27431.

79.

Marschinski

Kantz

Analysing the information flow between financial time series. Eur Phys J B Condense Matter Complex Syst 2002; 30: 275–281.

80.

Chowdhury

Stauffer

A generalized spin model of financial markets. Eur Phys J B Condense Matter Complex Syst 1999; 8: 477–482.

81.

Ghashghaie

Breymann

Peinke

et al . Turbulent cascades in foreign exchange markets. Nature 1996; 381: 767.

82.

Plerou

Gopikrishnan

Rosenow

et al . Universal and nonuniversal properties of cross correlations in financial time series. Phys Rev Lett 1999; 83: 1471.

83.

Lux

Marchesi

Scaling and criticality in a stochastic multi-agent model of a financial market. Nature 1999; 397: 498.

84.

Sornette

Johansen

Bouchaud

JP.

Stock market crashes, precursors and replicas. J Phys I 1996; 6: 167–175.

85.

Mantegna

Stanley

HE.

Introduction to econophysics: correlations and complexity in finance. Cambridge: Cambridge University Press, 1999.

86.

Kwon

Yang

JS.

Information flow between composite stock index and individual stocks. Phys A Stat Mech Appl 2008; 387: 2851–2856.

87.

Hong

White

Asymptotic distribution theory for nonparametric entropy measures of serial dependence. Econometrica 2005; 73: 837–901.

88.

Joe

Relative entropy measures of multivariate dependence. J Am Stat Assoc 1989; 84: 157–164.

89.

Dimitriadis

Sun

Laskaris

et al . Revealing cross-frequency causal interactions during a mental arithmetic task through symbolic transfer entropy: a novel vector-quantization approach. IEEE T Neural Syst Rehabil Eng 2016; 24: 1017–1028.

90.

Porta

Faes

Nollo

et al . Conditional self-entropy and conditional joint transfer entropy in heart period variability during graded postural challenge. PLoS ONE 2015; 10: e0132851.

91.

Murari

Peluso

Gelfusa

et al . Application of transfer entropy to causality detection and synchronization experiments in tokamaks. Nucl Fusion 2015; 56: 026006.

92.

Dickten

Lehnertz

Identifying delayed directional couplings with symbolic transfer entropy. Phys Rev E 2014; 90: 062706.

93.

Sensoy

Sobaci

Sensoy

et al . Effective transfer entropy approach to information flow between exchange rates and stock markets. Chaos Soliton Fract 2014; 68: 180–185.

94.

Zunino

Zanin

Tabak

et al . Complexity-entropy causality plane: a useful approach to quantify the stock market inefficiency. Phys A Stat Mech Appl 2010; 389: 1891–1901.

95.

Bachelier

Théorie de la spéculation. Annales scientifiques de l’École normale supérieure. Vol. 17, 1900.

96.

De Gregorio

Iacus

. On Rényi information for ergodic diffusion processes. Inform Sci 2009; 179: 279–291.

97.

Dionisio

Menezes

Mendes

DA.

An econophysics approach to analyse uncertainty in financial markets: an application to the Portuguese stock market. Eur Phys J B Condense Matter Complex Syst 2006; 50: 161–164.

98.

Matesanz

Ortega

GJ.

A (econophysics) note on volatility in exchange rate time series: entropy as a ranking criterion. Int J Modern Phys C 2008; 19: 1095–1103.

99.

Kim

Eom

Market efficiency in foreign exchange markets. Phys A: Stat Mech Appl 2007; 382: 209–212.

100.

Pincus

Kalman

RE.

Irregularity, volatility, risk, and financial market time series. Proc Natl Acad Sci U S A 2004; 101: 13709–13714.

101.

Rukhin

AL.

Approximate entropy for testing randomness. J Appl Prob 2000; 37: 88–100.

102.

Risso

WA.

The informational efficiency and the financial crashes. Res Int Bus Finance 2008; 22: 396–408.

103.

Zunino

Zanin

Tabak

et al . Forbidden patterns, permutation entropy and stock market inefficiency. Phys A Stat Mech Appl 2009; 388: 2854–2864.

104.

Xiong

Does the singular value decomposition entropy have predictive power for stock market?—evidence from the Shenzhen stock market. Phys A Stat Mech Appl 2015; 439: 103–113.

105.

Brooks

Persand

Volatility forecasting for risk management. J Forecast 2003; 22: 1–22.

106.

Giot

Laurent

Modelling daily value-at-risk using realized volatility and ARCH type models. J Empir Finance 2004; 11: 379–398.

107.

Caraiani

The predictive power of singular value decomposition entropy for stock market dynamics. Phys A Stat Mech Appl 2014; 393: 571–578.

108.

Barnett

Barrett

Seth

AK.

Granger causality and transfer entropy are equivalent for Gaussian variables. Phys Rev Lett 2009; 103: 238701.

109.

Gourieroux

Monfort

Renault

Kullback causality measures. Ann Econ Stat 1987; 6–7: 369–410.

110.

Granger

CW.

Testing for causality: a personal viewpoint. J Econ Dynam Control 1980; 2: 329–352.

111.

Darbellay

Wuertz

The entropy as a tool for analysing statistical dependences in financial time series. Phys A Stat Mech Appl 2000; 287: 429–439.

112.

Powell

Percival

IC.

A spectral entropy method for distinguishing regular and irregular motion of Hamiltonian systems. J Phys A Math Gen 1979; 12: 2053.

113.

Daw

Finney

Tracy

ER.

A review of symbolic analysis of experimental data. Rev Sci Instrum 2003; 74: 915–930.

114.

Broock

Scheinkman

Dechert

et al . A test for independence based on the correlation dimension. Econ Rev 1996; 15: 197–235.

115.

Cameron

Windmeijer

FA.

An R-squared measure of goodness of fit for some common nonlinear regression models. J Econometrics 1997; 77: 329–342.

116.

De Micco

González

Larrondo

et al . Randomizing nonlinear maps via symbolic dynamics. Phys A Stat Mech Appl 2008; 387: 3373–3383.

117.

Shu

Zeng

Chen

et al . Performance assessment of kernel density clustering for gene expression profile data. Int J Genomics 2003; 4: 287–299.

118.

Rosso

Mairal

ML.

Characterization of time dynamical evolution of electroencephalographic epileptic records. Phys A Stat Mech Appl 2002; 312: 469–504.

119.

Rosso

Blanco

Yordanova

et al . Wavelet entropy: a new tool for analysis of short duration brain electrical signals. J Neurosci Methods 2001; 105: 65–75.

120.

Huang

Shang

Zhao

Multifractal diffusion entropy analysis on stock volatility in financial markets. Phys A Stat Mech Appl 2012; 391: 5739–5745.

121.

Muzy

Bacry

Arneodo

The multifractal formalism revisited with wavelets. Int J Bifu Chaos 1994; 4: 245–302.

122.

Muzy

Bacry

Arneodo

Wavelets and multifractal formalism for singular signals: application to turbulence data. Phys Rev Lett 1991; 67: 3515.

123.

Peng

Buldyrev

Havlin

et al . Mosaic organization of DNA nucleotides. Phys Rev E 1994; 49: 1685.

124.

Kantelhardt

Zschiegner

Koscielny-Bunde

et al . Multifractal detrended fluctuation analysis of nonstationary time series. Phys A Stat Mech Appl 2002; 316: 87–114.

125.

Scafetta

Grigolini

Scaling detection in time series: diffusion entropy analysis. Phys Rev E 2002; 66: 036130.

126.

Scafetta

Latora

Grigolini

Lévy scaling: the diffusion entropy analysis applied to DNA sequences. Phys Rev E 2002; 66: 031906.

127.

Cai

Zhou

Yang

et al . Diffusion entropy analysis on the scaling behavior of financial markets. Phys A Stat Mech Appl 2006; 367: 337–344.

128.

Shi

Shang

Cross-sample entropy statistic as a measure of synchronism and cross-correlation of stock markets. Nonlin Dynam 2013; 71: 539–554.

129.

Anderson

PW.

The economy as an evolving complex system. Boca Raton, FL: CRC Press, 2018.

130.

Laloux

Cizeau

Bouchaud

et al . Noise dressing of financial correlation matrices. Phys Rev Lett 1999; 83: 1467.

131.

Amritkar

RE.

Stochastic dynamical model for stock-stock correlations. Phys Rev E 2004; 70: 026101.

132.

Stanley

Mantegna

RN.

An introduction to econophysics. Cambridge: Cambridge University Press, 2000.

133.

Bouchaud

JP.

Elements for a theory of financial risks. Phys A Stat Mech Appl 1999; 263: 415–426.

134.

Mandelbrot

BB.

Scaling in financial prices: II. Multifractals and the star equation. Quant Finance 2001; 1: 124–130.

135.

Kristoufek

Vosvrda

Measuring capital market efficiency: long-term memory, fractal dimension and approximate entropy. Eur Phys J B 2014; 87: 162.

136.

Giada

Marsili

Algorithms of maximum likelihood data clustering with applications. Phys A Stat Mech Appl 2002; 315: 650–664.

137.

Liu

Qian

HY.

Cross-sample entropy of foreign exchange time series. Phys A Stat Mech Appl 2010; 389: 4785–4792.

138.

Pincus

Approximate entropy (ApEn) as a complexity measure. Chaos 1995; 5: 110–117.

139.

Pincus

Approximate entropy as an irregularity measure for financial data. Econ Rev 2008; 27: 329–362.

140.

Pincus

SM.

Approximate entropy as a measure of system complexity. Proc Natl Acad Sci U S A 1991; 88: 2297–2301.

141.

Pincus

Huang

WM.

Approximate entropy: statistical properties and applications. Commun Stat Theory Methods 1992; 21: 3061–3077.

142.

Schwarz

Estimating the dimension of a model. Ann Stat 1978; 6: 461–464.

A review of entropy measures for uncertainty quantification of stochastic processes

Abstract

Keywords

Introduction

Conditional heteroscedastic models

Entropy measures for uncertainty quantification

Shannon entropy

Renyi and Tsallis entropies

Sample entropy

Step 1

Step 2

Step 3

Step 4

Mutual information and transfer entropy

Effective transfer entropy

Entropy rate

Normalized permutation entropy and number of forbidden patterns

Entropy measures based on singular value decomposition

Approximate entropy

Case study

Discussion and conclusion

Footnotes

Declaration of conflicting interests

Funding

ORCID iDs

References