Distributed Channel-Aware Quantization Based on Maximum Mutual Information

Abstract

In distributed sensing systems with constrained communication capabilities, sensors' noisy measurements must be quantized locally before transmitting to the fusion centre. When the same parameter is observed by a number of sensors, the local quantization rules must be jointly designed to optimize a global objective function. In this work we jointly design the local quantizers by maximizing the mutual information as the optimization criterion, so that the quantized measurements carry the most information about the unknown parameter. A low-complexity iterative approach is suggested for finding the local quantization rules. Using the mutual information as the design criterion, we can easily integrate the effect of communication channels in the design and consequently design channel-aware quantization rules. We observe that the optimal design depends on both the measurement and channel noises. Moreover, our algorithm can be used to design quantizers that can be deployed in different applications. We demonstrate the success of our technique through simulating estimation and detection applications, where our method achieves estimation and detection errors as low as when designing for those special purposes.

1. Introduction

A random source with continuous amplitude requires infinite number of bits to be described. However, due to practical constraints in communication systems, for example, limited storage or channel capacity, only finite number of bits can be accommodated. To compress a continuous-amplitude random source into limited amount of information its amplitude X should be quantized to $Q(X)$ which takes values from a discrete finite set. A side effect of this quantization is some loss of information about X, which depends on the quantizer's quality and the compression rate.

Quantization theory has been studied long ago [1, 2]. The rate-distortion theory [3] describes the relation between the amount of distortion caused by the quantization and the rate by which the quantized source can be presented. The theoretical limits described by the rate-distortion function can only be asymptotically achieved by optimal source encoding. For designing optimal quantizers, a practical method has been investigated by Lloyd and Max [1, 4]. They propose an iterative algorithm for finding the best quantization rule for a random source to achieve the lowest distortion. The distortion measure they use is the mean squared error (MSE). Using the Lloyd-Max algorithm, the optimal (it must be mentioned that all iterative algorithms for quantization design find a local optimal solution which depends on the initial quantization rules) $\hspace{0.17em}\hspace{0.17em}L$ -level quantization rule which minimizes the estimation MSE can be found for a random scalar X distributed according to probability density function (pdf) $p(X).$ The joint quantization of multiple variables has been studied under vector quantization [5, 6].

A more interesting quantization scenario is when the continuous-amplitude source is not observable and only a noisy version of it can be measured as $Y=X+W$ , where W is some random noise. In this scenario the observed quantity is quantized as $Q(Y)$ ; however, the goal is to achieve the best representation of X. It has been shown by [7–9] that the optimal quantizer in this case can be achieved using the generalized Lloyd-Max algorithm, where the distortion measure is modified to include both the quantization and measurement noises.

A relatively recent and more challenging problem appears in distributed sensing systems, for example, sensor networks. The problem can be described as distributed noisy source quantization, also referred to as the multiterminal source coding or CEO problem [10, 11]. In a distributed sensing system, the same unknown source is observed by different measurement devices, each having a noisy observation as $Y_n=X+W_n;\hspace{0.17em}\hspace{0.17em}\mathrm{1}\le n\le N$ , where $W_n$ is the measurement noise of the nth observation. Each observation has to be quantized according to a local quantization rule $Q_n(Y_n)$ and sent to the fusion centre (FC). At the FC, the quantized values are used for estimation, detection, or classification purposes, based on the application of the sensor network. Since for an optimal solution the local quantization rules have to be jointly optimized, this problem is more challenging than a centralized quantization design. The rate-distortion bound is analytically intractable in this case; however, upper and lower bound have been derived in [12].

Design of the optimal distributed quantizer for the above scenario has been considered by the authors of [13–18], who suggest cyclic algorithms based on alternating minimization [19], to find the optimal N quantization rules. The algorithm starts by initial guesses about the N quantizers, that is, $Q_{\mathrm{1}}^{\mathrm{0}},\dots ,Q_N^{\mathrm{0}}$ . During each iteration j, for each $\mathrm{1}\le n\le N$ , the best quantization rule that optimizes a performance criterion, for example, MSE [13, 14], Fisher information [15], or Ali-Silvey distances [16], is found by fixing the other $N-\mathrm{1}$ quantizers and is assigned to $Q_n^j$ .

Compared with the previous design algorithms for distributed quantization, that is, [13–16], in this work we use the mutual information (MI) as the optimization criterion for the distributed quantization design. We jointly design quantizers that maximize the MI of the quantized data and the unknown parameter. Our motivation for using the MI is that it is a fundamental measure showing how much information one variable contains about another variable. We design a set of quantizers for the noisy measurements, in a way that the quantized variables contain the most information about the unknown parameter.

A theory of designing quantizers based on optimizing mutual information measures has been discussed as the information bottleneck method by [17]. The information bottleneck method has been mostly used in clustering and classification applications. In this work we take the channel noise effect into account and design distributed quantizers that are optimum in presence of imperfect communication channels.

The MI measure has the following benefits. It allows designing the quantizers independent of the choice of a decoder or estimator in the FC. Also, as we will discuss later, when using the MI measure the global optimization criterion can be broken down into smaller criteria. Finally, it allows incorporating the effect of communication channels in the design of optimal quantizers. Hence, obtain the optimal distributed channel-aware quantizer. By maximizing the MI of the received data at the FC and the unknown parameter, we observe that depending on the channel noise the optimal quantizers can be different from the channel-unaware quantizers.

Performance evaluation through simulating different scenarios shows great results for our distributed quantizers design based on MI maximization. This is evaluated for two applications, that is, estimation and detection. We will show that the quantization rules obtained by maximizing the MI achieve the same, and in some cases better, performance when compared with those quantizers specifically designed for the estimation or the detection purpose, that is, using MSE or Ali-Silvey distances as the optimization criteria.

The paper is organized based on two cases. First, we assume perfect communication channels between the sensors and the FC and develop our method; then we apply the method to the general case where channel is not perfect. In Section 2, we first justify the choice of MI to be used as the design criterion. In Section 3, the problem is defined and formulated based on MI. Consequently, a design algorithm is devised in Section 4 assuming ideal communication channels. In Section 5, the algorithm is modified to include channel effect. Finally, the numerical results are presented and discussed in Section 6.

2. Mutual Information as the Optimization Criterion

Most of the literature on optimal quantizer design has used distortion measures, such as MSE, to design the optimal quantizer [1, 4, 7–9, 13]. However, other measures have also been used as criteria to design quantizers; among them are Ali-Silvey distances [16, 20, 21], Cramer-Rao lower bound, and Fisher information [15, 22, 23]. The motivation for using these measures is the fact that they work better in some applications. For example, Ali-Silvey distance measures are shown to design better quantizers for detection applications [16, 20].

A fundamental measure, showing how much information about the unknown is conveyed in the quantized data, is the MI of the unknown and the quantized data. Therefore, in this work, we base the design of distributed quantizers on maximizing the MI and will show, in Section 6, that the MI criteria result in designing quantizers with the same and even higher performance than other measures including distortion measures and Fisher information. Also, choice of MI as the optimization measure has some computation benefits in solving joint optimization of quantizers as it allows for breaking down the global bigger equation into smaller ones, as explained in Section 3. Keeping the number of quantized levels per sensor constant, we achieve the highest information rate $R=I(X;\widehat{X})$ by properly designing the quantizers. The MI criterion, to the best of our knowledge, has not been studied for distributed quantizer design in the literature.

A benefit of using the MI is to make the quantizer design independent of the estimation method or decoder. In design solutions based on distortion measures, such as squared error or Hamming error [3], the estimation method is fixed, for example, minimum mean squared error (MMSE) or maximum likelihood, and the optimization of the quantizers is achieved depending on the estimator type. Using the MI measure, however, following the design of quantizers, an estimation or detection method can be developed in the FC, based on each specific application. This enables us to design a quantizer that is useful for estimation, detection, classification, or feature extraction. Specifically for estimation purposes, the optimal quantizers designed based on minimizing the MSE (the Lloyd-Max algorithm) are those also with high MI [1, 24]. This makes sense, because when the quantized data carry more information about the unknown parameter, the FC has a better representation of the unknown; hence, it can estimate it more accurately. The performance of our MI-based algorithm in estimation and detection applications is discussed in Sections 6.1 and 6.2, respectively.

Using the MI measure in the distributed quantization design enables breaking down an N-sensor quantization problem into smaller problems. In fact, since the formula of the MI can be recursively broken down using the chain rule of MI, a simpler suboptimal solution can be derived by maximizing each component. The related formulations are discussed in the following section.

3. Problem Formulation Based on Mutual Information

The distributed quantization problem addressed in this work is defined as follows.

Suppose X   (for the brevity of notations, we use the same symbol to address a random variable and its value) is a random scalar which takes values in $\mathbb{R}$ with pdf $p(X)$ . A number of noisy measurements of X are observed at some distributed locations as $Y_n=h\left(X,W_n\right),\hspace{0.17em}\hspace{0.17em}\mathrm{1}\le n\le N$ , where h is the measurement function and $W_n$ is the measurement noise. The measurement noise at different sensors may be correlated, but it is assumed that the distribution $p(Y_{\mathrm{1}},\dots ,Y_N\mid X)$ is known.

Due to communication constraints, the continuous-amplitude measurements have to be quantized before transmission. Therefore, $Y_n$ is encoded to $Z_n\in {\mathcal{L}}_n=\{\mathrm{1,2},\dots ,L_n\}$ using a local quantization rule ${\mathcal{Q}}_n:\mathbb{R}\to {\mathcal{L}}_n$ ; that is, $Z_n={\mathcal{Q}}_n(Y_n)$ . A quantization rule ${\mathcal{Q}}_n$ is defined by a set of real-valued numbers called breakpoints, that is, ${\Gamma }_n$ , that divide $\mathbb{R}$ into partitions and assign a value from ${\mathcal{L}}_n$ to each partition. Each quantized piece of data $Z_n$ , $\mathrm{1}\le n\le N$ , is then transmitted over a communication channel, and the received symbol at the FC is called $T_n$ . The complete problem model is shown in Figure 1.

Figure 1

The complete model of the problem.

Let $\{Y_{\mathrm{1}},\dots ,Y_N\}$ , $\{Z_{\mathrm{1}},\dots ,Z_N\}$ , and $\{T_{\mathrm{1}},\dots ,T_N\}$ be denoted by $Y_{\mathrm{1}:N}$ , $Z_{\mathrm{1}:N}$ , and $T_{\mathrm{1}:N}$ , respectively. Then, random variables X, $Y_{\mathrm{1}:N}$ , $Z_{\mathrm{1}:N}$ , and $T_{\mathrm{1}:N}$ make a first-order Markov chain as $X\to Y_{\mathrm{1}:N}\to Z_{\mathrm{1}:N}\to T_{\mathrm{1}:N}$ ; hence

$$\begin{array}{cc}& P\left(X,Y_{\mathrm{1}:N},Z_{\mathrm{1}:N},T_{\mathrm{1}:N}\right)=P\left(X\right)P\left(Y_{\mathrm{1}:N}\mid X\right)P\left(Z_{\mathrm{1}:N}\mid Y_{\mathrm{1}:N}\right)P\left(T_{\mathrm{1}:N}\mid Z_{\mathrm{1}:N}\right).\end{array}$$

(1)

We will use this property to develop the design algorithm in Sections 4 and 5. We first consider ideal communication channels between the sensors and the FC and derive an algorithm for designing the quantization rules in Section 4. In Section 5, we extend the algorithm to consider the channel effect and design channel-aware quantization rules.

From this point to Section 5, the communication channels are assumed to be ideal. The goal is to derive ${\mathcal{Q}}_{\mathrm{1}}\left(·\right),\dots ,{\mathcal{Q}}_N(·)$ so that on average the random variables $Z_{\mathrm{1}},\dots ,Z_N$ together are a better representation of X. To achieve that, we maximize the MI of X and the N quantized data over N local quantization rules, that is,

$$\begin{array}{cc}& \underset{{\mathcal{Q}}_{\mathrm{1}},\dots ,{\mathcal{Q}}_N}{\max }I\left(X;Z_{\mathrm{1}},Z_{\mathrm{2}},\dots ,Z_N\right).\end{array}$$

(2)

Due to the stepwise characteristic of the quantization rules, finding an analytical solution to the problem in (2) is difficult. Therefore, in this work we find some approximations and numerical methods to tackle this problem.

The MI in (2) can be recursively written based on the chain rule of mutual information [3]:

$$\begin{array}{cc}& I\left(X;Z_{\mathrm{1}},Z_{\mathrm{2}},\dots ,Z_N\right)={\displaystyle \sum }_{m=\mathrm{1}}^{N}I\left(X;Z_m\mid Z_{m-\mathrm{1}},\dots ,Z_{\mathrm{1}}\right).\end{array}$$

(3)

Hence, a suboptimal solution can be derived based on this recursive breakdown as

$$\begin{array}{cc}& {\mathcal{Q}}_{\mathrm{1}}^{\ast }=\mathrm{a}\mathrm{r}\mathrm{g}\underset{{\mathcal{Q}}_{\mathrm{1}}}{\hspace{0.17em}\max }\hspace{0.17em}I\left(X;Z_{\mathrm{1}}\right),\\ {\mathcal{Q}}_{\mathrm{2}}^{\ast }=\mathrm{a}\mathrm{r}\mathrm{g}\underset{{\mathcal{Q}}_{\mathrm{2}}}{\hspace{0.17em}\max }\hspace{0.17em}I\left(X;Z_{\mathrm{2}}\mid Z_{\mathrm{1}}\right),\\ ⋮\\ {\mathcal{Q}}_n^{\ast }=\mathrm{a}\mathrm{r}\mathrm{g}\underset{{\mathcal{Q}}_n}{\hspace{0.17em}\max }\hspace{0.17em}I\left(X;Z_n\mid Z_{\mathrm{1}},Z_{\mathrm{2}},\dots ,Z_{n-\mathrm{1}}\right),\\ ⋮\\ {\mathcal{Q}}_N^{\ast }=\mathrm{a}\mathrm{r}\mathrm{g}\underset{{\mathcal{Q}}_N}{\hspace{0.17em}\max }\hspace{0.17em}I\left(X;Z_N\mid Z_{\mathrm{1}},Z_{\mathrm{2}},\dots ,Z_{N-\mathrm{1}}\right),\end{array}$$

(4)

where in the nth line, $\mathrm{1}\le n\le N$ , the quantized data $Z_{\mathrm{1}},\dots ,Z_{n-\mathrm{1}}$ are generated based on the previously found ${\mathcal{Q}}_{\mathrm{1}}^{\ast },\dots ,{\mathcal{Q}}_{n-\mathrm{1}}^{\ast }$ , respectively.

Finding the nth quantization rule, for $\mathrm{1}\le n\le N$ , from (4) is less complex than finding all quantization rules from (2). Therefore, this suboptimal approach has less complexity. In the following section, we develop a method to find a solution for the maximization problem in (4).

4. Design Algorithm

To find ${\mathcal{Q}}_n$ , $\mathrm{1}\le n\le N$ , one should solve the maximization in the nth line of (4). However, since ${\mathcal{Q}}_n$ is a discrete level function the optimization is not analytically traceable. In this section we provide a numerical method to find a local optimal solution for the nth quantization rule, $\mathrm{1}\le n\le N$ . In (4), assume that ${\mathcal{Q}}_{\mathrm{1}}^{\ast },\dots ,{\mathcal{Q}}_{n-\mathrm{1}}^{\ast }$ are known. To find the nth quantization rule ${\mathcal{Q}}_n^{\ast }$ , according to (4) we need to maximize $I(X;Z_n\mid Z_{\mathrm{1}},Z_{\mathrm{2}},\dots ,Z_{n-\mathrm{1}})$ . When the $n-\mathrm{1}$ previous quantization rules are fixed, this is equivalent to the following optimization (note that $Z_{\mathrm{1}:n}=\{Z_{\mathrm{1}},\dots ,Z_n\}$ ):

$$\begin{array}{cc}& {\mathcal{Q}}_n^{\ast }=\mathrm{a}\mathrm{r}\mathrm{g}\underset{{\mathcal{Q}}_n}{\hspace{0.17em}\max }\hspace{0.17em}I\left(X;Z_{\mathrm{1}:n}\right).\end{array}$$

(5)

Using the chain rule of MI and the definition of MI based on the entropy [3],

$$\begin{array}{cc}& I\left(X;Z_{\mathrm{1}:n}\right)=I\left(X,Y_{\mathrm{1}:n};Z_{\mathrm{1}:n}\right)-I\left(Y_{\mathrm{1}:n};Z_{\mathrm{1}:n}\mid X\right)=H\left(X,Y_{\mathrm{1}:n}\right)-H\left(X,Y_{\mathrm{1}:n}\mid Z_{\mathrm{1}:n}\right)-H\left(Y_{\mathrm{1}:n}\mid X\right)+H\left(Y_{\mathrm{1}:n}\mid X,Z_{\mathrm{1}:n}\right).\end{array}$$

(6)

In (6), the entropy terms involving only X and $Y_{\mathrm{1}:n}$ are independent of the choice of quantizers. Therefore, we can reduce the optimization problem to (we have dropped $dX,d{Y}_{\mathrm{1}},\dots ,d{Y}_n$ from all the formulas, to save space)

$$\begin{array}{cc}& {\mathcal{Q}}_n^{\ast }=\mathrm{a}\mathrm{r}\mathrm{g}\underset{{\mathcal{Q}}_n}{\hspace{0.17em}\max }\hspace{0.17em}\left[-H\left(X,Y_{\mathrm{1}:n}\mid Z_{\mathrm{1}:n}\right)+H\left(Y_{\mathrm{1}:n}\mid X,Z_{\mathrm{1}:n}\right)\right]=\mathrm{a}\mathrm{r}\mathrm{g}\underset{{\mathcal{Q}}_n}{\hspace{0.17em}\max }\hspace{0.17em}{{\displaystyle \int }}_X{{\displaystyle \int }}_{Y_{\mathrm{1}:n}}{\displaystyle \sum }_{Z_{\mathrm{1}:n}}p\left(X,Y_{\mathrm{1}:n},Z_{\mathrm{1}:n}\right)\log p\left(X\mid Z_{\mathrm{1}:n}\right)=\mathrm{a}\mathrm{r}\mathrm{g}\underset{{\mathcal{Q}}_n}{\hspace{0.17em}\max }\hspace{0.17em}{{\displaystyle \int }}_X{{\displaystyle \int }}_{Y_{\mathrm{1}:n}}{\displaystyle \sum }_{Z_{\mathrm{1}:n}}p\left(X,Y_{\mathrm{1}:n}\right)p\left(Z_{\mathrm{1}:n}\mid Y_{\mathrm{1}:n}\right)\log p\left(X\mid Z_{\mathrm{1}:n}\right),\end{array}$$

(7)

where the last equation is the consequence of the Markov chain property in (1). Note that since, for each $\mathrm{1}\le m\le n$ , $Z_m$ solely depends on $Y_m$ , we have

$$\begin{array}{cc}& p\left(Z_{\mathrm{1}:n}\mid Y_{\mathrm{1}:n}\right)=p\left(Z_{\mathrm{1}}\mid Y_{\mathrm{1}}\right)\times p\left(Z_{\mathrm{2}}\mid Y_{\mathrm{2}}\right)\cdots p\left(Z_n\mid Y_n\right).\end{array}$$

(8)

Therefore, (7) can also be written as

$$\begin{array}{cc}& {\mathcal{Q}}_n^{\ast }=\mathrm{a}\mathrm{r}\mathrm{g}\underset{p\left(Z_n\mid Y_n\right)}{\hspace{0.17em}\max }\hspace{0.17em}{{\displaystyle \int }}_X{{\displaystyle \int }}_{Y_{\mathrm{1}:n}}{\displaystyle \sum }_{Z_{\mathrm{1}:n}}p\left(X,Y_{\mathrm{1}:n}\right)\times p\left(Z_{\mathrm{1}:n-\mathrm{1}}\mid Y_{\mathrm{1}:n-\mathrm{1}}\right)p\left(Z_n\mid Y_n\right)\log p\left(X\mid Z_{\mathrm{1}:n}\right).\end{array}$$

(9)

Note that, in the above formula, the maximization is on $p(Z_n\mid Y_n)$ . It is straightforward to see that the probability function $p(Z_n\mid Y_n)$ is just another form of defining the nth quantization rule. Since, $\forall y\in \mathbb{R}$ , ${\mathcal{Q}}_n$ maps y to a value $l\in {\mathcal{L}}_n$ such that $l=Q_n(y)$ , we can write

$$\begin{array}{cc}& p\left(Z_n\mid Y_n=y\right)={\delta }_{l{Z}_n},\end{array}$$

(10)

where ${\delta }_{ab}$ is the Kronecker delta function, which is equal to one, where $a=b$ and zero elsewhere. Note that, in (9), $p(X\mid Z_{\mathrm{1}:n})$ also depends on the quantization rule or $p(Z_n\mid Y_n)$ .

To solve the optimization problem in (9), motivated by [25] we use the double maxima approach by converting (9) to a larger maximization problem. The maximization in (9) can be achieved following the next three steps, which is proven in the Appendix.

• (i)

The maximum of the objective function in (9), namely, $I^{\ast }$ , can be written as

$$\begin{array}{cc}& I^{\ast }=\underset{p}{\max }\underset{f}{\hspace{0.17em}\max }\hspace{0.17em}\hspace{0.17em}I\left(p,f\right),\end{array}$$

(11)

where p is a short term for $p(Z_n\mid Y_n)$ and f is a short term for $f(X,Z_{\mathrm{1}:n})$ , a function of X and $Z_{\mathrm{1}:n}$ , such that, for all realizations of $Z_{\mathrm{1}:n}\in {\mathcal{L}}_{\mathrm{1}}\times {\mathcal{L}}_{\mathrm{2}}\times \cdots \times {\mathcal{L}}_n$ , ${\sum }_X‍f(X,Z_{\mathrm{1}:n})=\mathrm{1}$ . And

$$\begin{array}{cc}& I\left(p,f\right)={{\displaystyle \int }}_X{{\displaystyle \int }}_{Y_{\mathrm{1}:n}}{\displaystyle \sum }_{Z_{\mathrm{1}:n}}p\left(X,Y_{\mathrm{1}:n}\right)\times p\left(Z_{\mathrm{1}:n-\mathrm{1}}\mid Y_{\mathrm{1}:n-\mathrm{1}}\right)p\left(Z_n\mid Y_n\right)\log f\left(X,Z_{\mathrm{1}:n}\right).\end{array}$$

(12)

(ii)

Now, for a fixed p, $I(p,f)$ is maximized by

$$\begin{array}{cc}& f^{\ast }\left(X,Z_{\mathrm{1}:n}\right)=\frac{{\int }_{Y_{\mathrm{1}:n}}p\left(X,Y_{\mathrm{1}:n}\right)P\left(Z_{\mathrm{1}:n}\mid Y_{\mathrm{1}:n}\right)}{{\int }_X{\int }_{Y_{\mathrm{1}:n}}p\left(X,Y_{\mathrm{1}:n}\right)P\left(Z_{\mathrm{1}:n}\mid Y_{\mathrm{1}:n}\right)}.\end{array}$$

(13)

(iii)

And, for a fixed f, $p^{\ast }$ that maximizes $I(p,f)$ is obtained as

$$\begin{array}{cc}& \forall y\in \mathbb{R},\hspace{1em}p\left(Z_n\mid Y_n=y\right)={\delta }_{l^{\ast }y},\\ \mathrm{w}\mathrm{h}\mathrm{e}\mathrm{r}\mathrm{e}\hspace{0.17em}\hspace{0.17em}{l}^{\ast }=\mathrm{a}\mathrm{r}\mathrm{g}\underset{l}{\hspace{0.17em}\max }\hspace{0.17em}{{\displaystyle \int }}_X{{\displaystyle \int }}_{Y_{\mathrm{1}:n-\mathrm{1}}}{\displaystyle \sum }_{Z_{\mathrm{1}:n-\mathrm{1}}}p\left(X,Y_{\mathrm{1}:n-\mathrm{1}},y\right)p\left(Z_{\mathrm{1}:n-\mathrm{1}}\mid Y_{\mathrm{1}:n-\mathrm{1}}\right)\log f\left(X,Z_{\mathrm{1}:n-\mathrm{1}},l\right),\end{array}$$

(14)

where $l\in {\mathcal{L}}_n=\{\mathrm{1,2},\dots ,L_n\}$ .

It is shown in the Appendix that these procedures find $I^{\ast }$ . Using (i), (ii), and (iii), an iterative algorithm to find the optimal nth quantization rule can be derived as Algorithm 1. In Algorithm 1, ϵ determines the condition to stop the iterations. Since the mutual information $I(X;Z_{\mathrm{1}:n})$ is increased during each iteration and it is upper-bounded, the algorithm converges to a local maximum.

Algorithm 1: Iterative algorithm to find nth quantization rule.

$j=\mathrm{0}$

initialize the quantization rule: $p^{\mathrm{0}}$

repeat

  $j=j+\mathrm{1}$

 maximize $I(p,f)$ for fixed p, according to (13)

       $$f^j=\underset{f}{\max }\hspace{0.17em}I\left(p^j,f\right)$$

 maximize $I(p,f)$ for fixed f, according to (22)

       $$p^{j+\mathrm{1}}=\underset{p}{\max }\hspace{0.17em}I\left(p,f^j\right)$$

until   $$\left|I\left(p^j,f^j\right)-I\left(p^{j-\mathrm{1}},f^{j-\mathrm{1}}\right)\right|>ϵI\left(p^j,f^j\right)$$

5. Channel-Aware Optimal Quantizers

The discussions up to this point have assumed ideal communication channels between the sensors and the FC. In real distributed sensing systems, due to the nonideal communication channels the quantized data generated by the sensors might not be received correctly at the FC. This will affect the overall performance of the system. Hence, considering the channel effect in designing the quantizers is crucial [26]. For centralized quantization [27–29], revise the MSE to include the channel effect. Then, they jointly optimize the source encoders and the reconstruction levels at the receiver by minimizing this new MSE. For distributed quantization, channel-optimized quantizer design has been developed for hypothesis testing by minimizing the Bayesian cost [30, 31]. Recently, the distributed channel-aware quantizer design for multiple correlated sources has been addressed by [32, 33], where M source encoders are designed to quantize M correlated sources, in presence of noisy communication channels. Reference [34] discusses the problem under a total power constraint and designs the quantizers by minimizing the signal distortion in the receiver. For multimedia applications in distributed networks, the multiple description coding has been used to fight the channel loss [33]. References [35–37] have addressed the effect of imperfect transmission channels in the multiple description coding algorithms for the application of distributed video transmission. In this section, we are considering the channel into our quantizer design to recover the signal more accurately in the destination.

In this section, we design optimal channel-aware quantizers for the distributed quantization of a noisy source using MI measure. We assume that communication channels between each piece of quantized data and the FC are independent. In presence of these noisy channels, we now optimize the quantizers' design by maximizing the MI of the unknown parameter and the channels' outputs. We use the Markov chain property in (1), and to solve the optimization problem we follow an approach similar to Section 4.

Due to channel errors, the received symbol at the FC, $T_n\in {\mathcal{L}}_n$ , might not be the same as the transmitted symbol $Z_n$ . We assume that the channel transition probabilities, that is, $p\left(T_n=l∣Z_n=k\right)$ , $\mathrm{1}\le l$ , $k\le L_n$ , are known. Based on these channel transition probabilities, we can write the MI of X and the received symbols at the channels' output $T_{\mathrm{1}},T_{\mathrm{2}},\dots ,T_N$ , that is, $I(X;T_{\mathrm{1}},T_{\mathrm{2}},\dots ,T_N)$ , or in short $I(X;T_{\mathrm{1}:N})$ . Then we maximize $I(X;T_{\mathrm{1}:N})$ to find the optimal N channel-aware quantizers. Similar arguments preceding (5) are applicable here. Hence, the nth optimal channel-aware quantization rule is obtained as

$$\begin{array}{cc}& {\mathcal{Q}}_n^{\ast }=\mathrm{a}\mathrm{r}\mathrm{g}\underset{{\mathcal{Q}}_n}{\hspace{0.17em}\max }\hspace{0.17em}I\left(X;T_{\mathrm{1}:n}\right).\end{array}$$

(15)

By substituting $T_{\mathrm{1}:n}$ instead of $Z_{\mathrm{1}:n}$ in (6), we can get

$$\begin{array}{cc}& {\mathcal{Q}}_n^{\ast }=\mathrm{a}\mathrm{r}\mathrm{g}\underset{{\mathcal{Q}}_n}{\hspace{0.17em}\max }\hspace{0.17em}I\left(X;T_{\mathrm{1}:n}\right)=\mathrm{a}\mathrm{r}\mathrm{g}\underset{{\mathcal{Q}}_n}{\hspace{0.17em}\max }\hspace{0.17em}{{\displaystyle \int }}_X{{\displaystyle \int }}_{Y_{\mathrm{1}:n}}{\displaystyle \sum }_{T_{\mathrm{1}:n}}p\left(X,Y_{\mathrm{1}:n}\right)p\left(T_{\mathrm{1}:n}\mid Y_{\mathrm{1}:n}\right)\log p\left(X\mid T_{\mathrm{1}:n}\right).\end{array}$$

(16)

Since $Y_{\mathrm{1}:n}$ , $Z_{\mathrm{1}:n}$ , and $T_{\mathrm{1}:n}$ make a Markov chain $Y_{\mathrm{1}:n}\to Z_{\mathrm{1}:n}\to T_{\mathrm{1}:n}$ , we have

$$\begin{array}{cc}& p\left(T_{\mathrm{1}:n}\mid Y_{\mathrm{1}:n}\right)={\displaystyle \sum }_{Z_{\mathrm{1}:n}}p\left(T_{\mathrm{1}:n}\mid Z_{\mathrm{1}:n}\right)p\left(Z_{\mathrm{1}:n}\mid Y_{\mathrm{1}:n}\right),\end{array}$$

(17)

and therefore

$$\begin{array}{cc}& {\mathcal{Q}}_n^{\ast }=\mathrm{a}\mathrm{r}\mathrm{g}\underset{p\left(Z_n\mid Y_n\right)}{\hspace{0.17em}\max }\hspace{0.17em}{{\displaystyle \int }}_X{{\displaystyle \int }}_{Y_{\mathrm{1}:n}}{\displaystyle \sum }_{Z_{\mathrm{1}:n}}\hspace{0.17em}{\displaystyle \sum }_{T_{\mathrm{1}:n}}p\left(X,Y_{\mathrm{1}:n}\right)p\left(Z_{\mathrm{1}:n}\mid Y_{\mathrm{1}:n}\right)p\left(T_{\mathrm{1}:n}\mid Z_{\mathrm{1}:n}\right)\log p\left(X\mid T_{\mathrm{1}:n}\right).\end{array}$$

(18)

Assuming that the channel between each sensor and the FC is independent of the other channels, $p(T_{\mathrm{1}:n}\mid Z_{\mathrm{1}:n})$ can be obtained as ${\prod }_{m=\mathrm{1}}^n‍p(T_m\mid Z_m)$ , where each term is the transition probability of the corresponding sensor-to-FC channel. Following the same steps as in Section 4, we can derive similar results. Note that the term $p(X\mid T_{\mathrm{1}:n})$ in (18) depends on the quantization rule $p(Z_n\mid Y_n)$ . First we write

$$\begin{array}{cc}& I_{\mathrm{c}\mathrm{h}}^{\ast }=\underset{p}{\max }\underset{f}{\hspace{0.17em}\max }\hspace{0.17em}{I}_{\mathrm{c}\mathrm{h}}\left(p,f\right),\end{array}$$

(19)

where p is a short term for $p(Z_n\mid Y_n)$ and f is a short term for $f(X,T_{\mathrm{1}:n})$ , such that, for all realizations of $T_{\mathrm{1}:n}\in {\mathcal{L}}_{\mathrm{1}}\times {\mathcal{L}}_{\mathrm{2}}\times \cdots \times {\mathcal{L}}_n$ , ${\sum }_X‍f(X,T_{\mathrm{1}:n})=\mathrm{1}$ . And

$$\begin{array}{cc}& I_c\left(p,f\right)={{\displaystyle \int }}_X{{\displaystyle \int }}_{Y_{\mathrm{1}:n}}{\displaystyle \sum }_{Z_{\mathrm{1}:n}}\hspace{0.17em}{\displaystyle \sum }_{T_{\mathrm{1}:n}}p\left(X,Y_{\mathrm{1}:n}\right)p\left(Z_{\mathrm{1}:n-\mathrm{1}}\mid Y_{\mathrm{1}:n-\mathrm{1}}\right)p\left(T_{\mathrm{1}:n}\mid Z_{\mathrm{1}:n}\right)p\left(Z_n\mid Y_n\right)\log f\left(X,T_{\mathrm{1}:n}\right).\end{array}$$

(20)

Then, for a fixed p, $I_{\mathrm{c}\mathrm{h}}(p,f)$ is maximized by

$$\begin{array}{cc}& f^{\ast }\left(X,T_{\mathrm{1}:n}\right)=\frac{{\int }_{Y_{\mathrm{1}:n}}{\sum }_{Z_{\mathrm{1}:n}}^{}p\left(X,Y_{\mathrm{1}:n}\right)p\left(Z_{\mathrm{1}:n}\mid Y_{\mathrm{1}:n}\right)p\left(T_{\mathrm{1}:n}\mid Z_{\mathrm{1}:n}\right)}{{\int }_X{\int }_{Y_{\mathrm{1}:n}}{\sum }_{Z_{\mathrm{1}:n}}p\left(X,Y_{\mathrm{1}:n}\right)p\left(Z_{\mathrm{1}:n}\mid Y_{\mathrm{1}:n}\right)p\left(T_{\mathrm{1}:n}\mid Z_{\mathrm{1}:n}\right)}.\end{array}$$

(21)

And, for a fixed f, $p^{\ast }$ that maximizes $I_c(p,f)$ is obtained as

$$\begin{array}{cc}& \forall y\in \mathbb{R},\hspace{1em}p\left(Z_n\mid Y_n=y\right)={\delta }_{l^{\ast }y},\\ \mathrm{w}\mathrm{h}\mathrm{e}\mathrm{r}\mathrm{e}\hspace{0.17em}\hspace{0.17em}{l}^{\ast }=\mathrm{a}\mathrm{r}\mathrm{g}\underset{l}{\hspace{0.17em}\max }\hspace{0.17em}{{\displaystyle \int }}_X{{\displaystyle \int }}_{Y_{\mathrm{1}:n-\mathrm{1}}}{\displaystyle \sum }_{Z_{\mathrm{1}:n-\mathrm{1}}}\hspace{0.17em}{\displaystyle \sum }_{T_{\mathrm{1}:n}}p\left(X,Y_{\mathrm{1}:n-\mathrm{1}},y\right)p\left(Z_{\mathrm{1}:n-\mathrm{1}}\mid Y_{\mathrm{1}:n-\mathrm{1}}\right)P\left(T_{\mathrm{1}:n-\mathrm{1}}\mid Z_{\mathrm{1}:n-\mathrm{1}}\right)p\left(T_n\mid Z_n=l\right)\log f\left(X,T_{\mathrm{1}:n}\right),\end{array}$$

(22)

where $l\in {\mathcal{L}}_n=\{\mathrm{1,2},\dots ,L_n\}$ . Finally, a similar iterative solution as Algorithm 1 can be proposed for finding N channel-aware quantization rules.

6. Simulation Results

In this section the performance of our proposed algorithm is demonstrated using computer simulations and compared with other methods. In particular, we examine the performance of our MI-based quantization design for the estimation applications and detection applications, in Sections 6.1 and 6.2, respectively. The effect of nonideal channels on the optimal quantization rules is investigated in Section 6.3.

6.1. Estimation Application

For a distributed sensing system with estimation purposes, the quantized values are used in the FC to estimate the unknown. To compare with [15], where the quantization rules are obtained by minimizing the MSE, we use a similar simulation scenario. Therefore, the unknown parameter X is distributed according to $p(X)=\mathcal{N}(\mathrm{0,1})$ . Two sensors are involved; that is, $N=\mathrm{2}$ . The measurement noises $W_{\mathrm{1}}$ and $W_{\mathrm{2}}$ are additive Gaussian noises with correlation ρ and marginal distribution $\mathcal{N}(\mathrm{0,1})$ . The number of quantization levels for both sensors is L. At the FC we use the MMSE estimator to estimate X from the quantized measurements $Z_{\mathrm{1}}$ and $Z_{\mathrm{2}}$ . Similar to [15], the initial quantization breakpoints are chosen from the optimal quantization rules of Lloy-Max algorithm [4]; that is, ${\Gamma }_{\mathrm{1}}^{\mathrm{0}}={\Gamma }_{\mathrm{2}}^{\mathrm{0}}=\{-\mathrm{2.5,0},\mathrm{2.5}\}$ .

Our algorithm finds the optimal quantizers ${\mathcal{Q}}_{\mathrm{1}}^{\ast }$ and ${\mathcal{Q}}_{\mathrm{2}}^{\ast }$ by maximizing the MI $I(X;Z_{\mathrm{1}},Z_{\mathrm{2}})$ . According to (4), $I(X;Z_{\mathrm{1}},Z_{\mathrm{2}})$ can be broken down as $I(X;Z_{\mathrm{1}})+I(X;Z_{\mathrm{2}}\mid Z_{\mathrm{1}})$ . Based on Algorithm 1, first $I(X;Z_{\mathrm{1}})$ is maximized to find ${\mathcal{Q}}_{\mathrm{1}}^{\ast }$ , and consequently $I(X;Z_{\mathrm{2}}\mid Z_{\mathrm{1}})$ is maximized to find ${\mathcal{Q}}_{\mathrm{2}}^{\ast }$ . Due to this breaking down of the task, the MI $I(X;Z_{\mathrm{1}},Z_{\mathrm{2}})$ , which is the sum of the two components, is maximized in two steps. Figure 2 shows the value of MI at each iteration of the algorithm, for $L=\mathrm{4}$ and $\rho =\mathrm{0}$ .

Figure 2

Designing the quantization rules by maximizing the MI.

At each iteration of the algorithm, the current quantization rules are used to quantize the measurements $Y_{\mathrm{1}}$ and $Y_{\mathrm{2}}$ to $Z_{\mathrm{1}}$ and $Z_{\mathrm{2}}$ , respectively. These values are then used to estimate X using the MMSE estimator; that is, $\widehat{X}=E\{X\mid Z_{\mathrm{1}},Z_{\mathrm{2}}\}$ . The estimation performance at each iteration is computed in terms of MSE, in Figure 3. It can be seen from Figures 2 and 3 that by maximizing the MI the MSE of estimation is decreased.

Figure 3

MSE change at each iteration.

The optimal quantization rules at the end of iterations are represented by the set of breakpoints as ${\Gamma }_{\mathrm{1}}$ and ${\Gamma }_{\mathrm{2}}$ in Table 1. For different simulation scenarios the final quantization rules and the final MSE are shown and compared with the results of Lam algorithm [15]. Comparing with [15], the final quantization rules are different, but the MSE performances are essentially the same.

Table 1

MSE for estimation.

$\rho =\mathrm{0}$ , $L=\mathrm{4}$
Method	Optimal quantization rules	MSE
MI	${\mathrm{\Gamma }}_{\mathrm{1}}={\mathrm{\Gamma }}_{\mathrm{2}}=\left\{\right.-\mathrm{1.37,0},\mathrm{1.37}\left.\right\}$	$\mathrm{0.3879}$
Lam	${\mathrm{\Gamma }}_{\mathrm{1}}={\mathrm{\Gamma }}_{\mathrm{2}}=\left\{\right.-\mathrm{1.39,0},\mathrm{1.39}\left.\right\}$	$\mathrm{0.3880}$
$\rho =\mathrm{0}$ , $L=\mathrm{8}$
Method	Optimal quantization rules	MSE
MI	${\mathrm{\Gamma }}_{\mathrm{1}}={\mathrm{\Gamma }}_{\mathrm{2}}=\left\{\right.-\mathrm{2.49},-\mathrm{1.51},-\mathrm{0.73,0},\mathrm{0.73,1.51,2.49}\left.\right\}$	$\mathrm{0.3484}$
Lam	${\mathrm{\Gamma }}_{\mathrm{1}}={\mathrm{\Gamma }}_{\mathrm{2}}=\left\{\right.-\mathrm{2.48},-\mathrm{1.48},-\mathrm{0.71,0},\mathrm{0.71,1.48,2.48}\left.\right\}$	$\mathrm{0.3487}$
$\rho =\mathrm{0.85}$ , $L=\mathrm{4}$
Method	Optimal quantization rules	MSE
MI	${\mathrm{\Gamma }}_{\mathrm{1}}=\left\{\right.-\mathrm{1.15,0},\mathrm{1.15}\left.\right\}$ , ${\mathrm{\Gamma }}_{\mathrm{2}}=\left\{\right.-\mathrm{2.10,0},\mathrm{2.10}\left.\right\}$	$\mathrm{0.5170}$
Lam	${\mathrm{\Gamma }}_{\mathrm{1}}=\left\{\right.-\mathrm{1.13,0},\mathrm{1.13}\left.\right\}$ , ${\mathrm{\Gamma }}_{\mathrm{2}}=\left\{\right.-\mathrm{2.08,0},\mathrm{2.08}\left.\right\}$	$\mathrm{0.5171}$

6.2. Detection Application

In a distributed sensing system with detection purposes, the FC uses the quantized data to perform a hypothesis testing. We use our method of maximizing the MI to find the optimal quantization rules for the detection scenario and compare the performance with that of Poor algorithm [20], where Ali-Silvey distances [38] are used as the optimization criterion.

To simulate the detection scenario we assume that the unknown X is a Bernoulli random variable which represents the absence ( $H_{\mathrm{0}}$ ) or presence ( $H_{\mathrm{1}}$ ) of the signal θ; that is, $P(X=\theta )=P_{\theta }$ and $P(X=\mathrm{0})=\mathrm{1}-P_{\theta }$ . Each sensor makes an observation of X in additive Gaussian noise and sends the quantized observation to the FC. We use the algorithm in Section 4 to design the optimal rules for quantizing the measurements. Note that since X takes its values from the finite set $\{\theta ,\mathrm{0}\}$ , the integral over X in all equations translates into a summation over this set. At the FC, the Neyman-Pearson method is used to test the hypotheses.

To compare with Poor [20], we assume equally likely $H_{\mathrm{0}}$ and $H_{\mathrm{1}}$ ; that is, $P_{\theta }=\mathrm{1}/\mathrm{2}$ . Also, each sensor quantizes its observation to $L=\mathrm{4}$ levels. And the measurement noises are i.i.d. with pdf $\mathcal{N}(\mathrm{0,1})$ . The probability of detection error is shown in Table 2 for two different signal energies. Our method is indicated by “MI” in the table. The results based on Matsushita distance and J-divergence criteria from [20] are also indicated in the table. It can be seen from the error probabilities that the detection performance is similar to and in some cases better than [20].

Table 2

Probability of error for signal detection.

		$\theta =\mathrm{2}$	$\theta =\mathrm{4}$
$N=\mathrm{5}$	MI	$\mathrm{1.7}\times \mathrm{1}{\mathrm{0}}^{-\mathrm{2}}$	$\mathrm{1.0}\times \mathrm{1}{\mathrm{0}}^{-\mathrm{5}}$
	Matsushita	$\mathrm{1.7}\times \mathrm{1}{\mathrm{0}}^{-\mathrm{2}}$	$\mathrm{1.1}\times \mathrm{1}{\mathrm{0}}^{-\mathrm{5}}$
	J-divergence	$\mathrm{1.7}\times \mathrm{1}{\mathrm{0}}^{-\mathrm{2}}$	$\mathrm{2.1}\times \mathrm{1}{\mathrm{0}}^{-\mathrm{5}}$

6.3. Channel Effect

The presence of a nonideal communication channel between each sensor and the FC affects the design of optimal local quantizers for each sensor. Using the design algorithm developed in Section 5, we find the channel-aware local quantizers. The simulation results confirm that the optimal quantizers assuming ideal channels are different from the optimal quantizers in the presence of nonideal channels.

To compare the channel-aware and channel-unaware quantization schemes we consider an estimation application. We assume that sensor n's quantized data $Z_n$ , $\mathrm{1}\le n\le N$ , is mapped to a binary word of size ${\mathrm{l}\mathrm{o}\mathrm{g}}_{\mathrm{2}}{L}_n$ and sent to the FC via a binary symmetric channel (BSC) with crossover probability ${ϵ}_n$ . The received symbol at the FC is therefore $T_n$ and the transition probabilities, that is, $p(T_n=l\mid Z_n=k)$ , $\mathrm{1}\le l$ , $k\le L_n$ , are derived based on ${ϵ}_n$ . In the simulation examples we assume two sensors with identical crossover probabilities for all channels; that is, ${ϵ}_{\mathrm{1}}={ϵ}_{\mathrm{2}}=ϵ$ .

Figures 4 and 5 show maximization of the MI and minimization of the MSE, respectively, during the iterations and for different values of ϵ. The final quantizers are given in Table 3. It can be seen from Table 3 that the optimal quantization solution changes depending on the channel error probability. Consequently, if, for instance, one deploys the quantizers designed for $ϵ=\mathrm{0}$ in a scenario where $ϵ=\mathrm{0.05}$ , the MSE will be $\mathrm{0.497}$ . While using the optimal quantizers designed for $ϵ=\mathrm{0.05}$ , the MSE is $\mathrm{0.485}$ .

Table 3

Optimal quantizers in presence of nonideal channel.

$\rho =\mathrm{0}$ , $L=\mathrm{4}$
$ϵ=\mathrm{0}$	${\mathrm{\Gamma }}_{\mathrm{1}}={\mathrm{\Gamma }}_{\mathrm{2}}=\left\{\right.-\mathrm{1.370,0},\mathrm{1.370}\left.\right\}$	MSE = $\mathrm{0.388}$
$ϵ=\mathrm{0.001}$	${\mathrm{\Gamma }}_{\mathrm{1}}={\mathrm{\Gamma }}_{\mathrm{2}}=\left\{\right.-\mathrm{1.358,0},\mathrm{1.358}\left.\right\}$	MSE = $\mathrm{0.391}$
$ϵ=\mathrm{0.01}$	${\mathrm{\Gamma }}_{\mathrm{1}}={\mathrm{\Gamma }}_{\mathrm{2}}=\left\{\right.-\mathrm{1.260,0},\mathrm{1.260}\left.\right\}$	MSE = $\mathrm{0.411}$
$ϵ=\mathrm{0.05}$	${\mathrm{\Gamma }}_{\mathrm{1}}={\mathrm{\Gamma }}_{\mathrm{2}}=\left\{\right.-\mathrm{0.973,0},\mathrm{0.973}\left.\right\}$	MSE = $\mathrm{0.485}$

Figure 4

Maximizing the MI in presence of nonideal communication channels.

Figure 5

MSE of estimation at each iteration of the algorithm.

For the error detection application in nonideal communication channels, we have compared our results with that of [31], for different problem setups. Reference [31] develops an iterative algorithm by minimizing the error probability at the fusion centre after finding the optimal fusion rule for each iteration based on the quantizers of that iteration. For all setups, we choose $N=\mathrm{2}$ , $L=\mathrm{4}$ , $\theta =\mathrm{1}$ , and i.i.d. measurement noises with pdf $\mathcal{N}(\mathrm{0,1})$ . ϵ and $P_{\theta }$ vary for different scenarios. The optimized quantizers and the final error probability are given in Table 4. It can be seen that for all cases our results are very close and just better than the results of [31].

Table 4

Error detection in presence of nonideal channel.

$N=\mathrm{2},L=\mathrm{4}$
Method	$P_{\theta }$	ϵ	${\mathrm{\Gamma }}_{\mathrm{1}}={\mathrm{\Gamma }}_{\mathrm{2}}$	MSE
Liu and Chen	0.5	0.1	$\left\{\right.-\mathrm{0.2009,0.4402,0.5778}\left.\right\}$	0.2932
MI	0.5	0.1	$\left\{\right.-\mathrm{0.0825,0.4125,0.7725}\left.\right\}$	0.2856
Liu and Chen	0.5	0.2	$\left\{\right.-\mathrm{0.0165,0.4168,0.4168}\left.\right\}$	0.3376
MI	0.5	0.2	$\left\{\right.-\mathrm{0.3525},-\mathrm{0.0275,0.1175}\left.\right\}$	0.3351
Liu and Chen	0.3	0.1	$\left\{\right.-\mathrm{0.1561,0.6098,0.6098}\left.\right\}$	0.2404
MI	0.3	0.1	$\left\{\right.\mathrm{0.0175,0.4775,0.8725}\left.\right\}$	0.2358
Liu and Chen	0.3	0.27	$\left\{\right.\mathrm{0.3115,0.3115,0.3115}\left.\right\}$	0.2855
MI	0.3	0.27	$\left\{\right.\mathrm{0.3325,0.6125,0.6125}\left.\right\}$	0.2756

7. Conclusion

In this paper, we proposed an algorithm based on maximizing the MI measure for jointly designing optimal channel-aware local quantization rules for a distributed sensing system. The MI allows us to design general purpose quantizers that later can be deployed for different applications, for example, estimation or detection. We have shown that the performance of the optimal quantizers based on the MI is essentially the same as the performance of optimal quantizers of other methods that specifically target the estimation or detection application. We also observed that the optimal local quantizers in the presence of nonideal channels are different from the local quantizers that are optimized without considering the channel effect.