Nonnegative Matrix Factorization Based Adaptive Noise Sensing over Wireless Sensor Networks

Abstract

An adaptive noise sensing method is proposed to improve the speech sensing performance of speech-based applications operated over wireless sensor networks. The proposed method is based on nonnegative matrix factorization (NMF), which consists of adaptive noise sensing and noise reduction. In other words, adaptive noise sensing is performed by adapting a priori noise basis matrix of the NMF, which is estimated from the noise signal, resulting in an adapted noise basis matrix. Subsequently, the adapted noise basis matrix is used for the NMF decomposition of noisy speech into clean speech and background noise. The estimated clean speech signal is then applied to a front-end of the speech-based applications. The performance of the proposed NMF-based noise sensing and reduction method is first evaluated by measuring the source to distortion ratio (SDR), the source to interferences ratio (SIR), and the source to artifacts ratio (SAR). In addition, the proposed method is applied to an automatic speech recognition (ASR) system, which is a typical speech-based application, and then the average word error rate (WER) of the ASR is compared with that employing either a Wiener filter, or a conventional NMF-based noise reduction method using only a priori noise basis matrix.

1. Introduction

Speech-based user applications operated over wireless sensor networks are increasingly being utilized in various environments, for example, smart home, smart TV, and cars, since they have become a key feature of smart user interfaces [1–4]. However, as the number of the speech-based application fields has increased, the various types of background noise could negatively affect the speech sensing performance of the applications deployed over wireless sensor networks. These background noises can be classified into two types—stationary and nonstationary—depending on the variability of their characteristics over time. Many conventional methods, including spectral subtraction [5], minimum mean square error log-spectral amplitude (MMSE-LSA) [6, 7], and Wiener filtering [8], have been reported to effectively reduce stationary noise that was recorded with speech signals. Consequently, they were successfully applied to a front-end of an automatic speech recognition (ASR) system, that is, a typical speech-based application over wireless sensor networks [9]. However, since these conventional methods were developed based on the stationary noise assumption, their performance could degrade under nonstationary noise conditions [10, 11]. Thus, the reduction of nonstationary noise is important for reliable noise-robust speech-based applications over wireless sensor networks under various kinds of environmental noise conditions.

As an alternative, nonnegative matrix factorization (NMF)-based noise reduction methods have been proposed to estimate the noise spectrum effectively under nonstationary noise conditions [12–16]. In particular, recent research works have reported that NMF-based noise reduction methods have been successfully applied to a front-end of an ASR system under various nonstationary noise environments [14–16]. However, the performance of NMF-based noise reduction methods degraded substantially when there was a mismatch in noise type for noise basis training and estimation using NMF [17, 18]. There have been several approaches proposed to improve the noise reduction performance of NMF when there was a mismatch between the training and estimation of speech and/or the noise basis [19–21]. In particular, real-time semisupervised source separation methods in [19, 20] assumed that the noise basis was prepared at the training stage, while the speech basis was learned online under nonstationary noisy conditions. The methods dealt with the mismatch in speech basis training and estimation, but there was no consideration on the mismatch between the noise basis training and estimation. In [21], a universal model was introduced to overcome the mismatch between the noise basis training and estimation, where each noise basis was trained to represent a certain type of noise source. Consequently, the performance of the universal model would be limited because noise could be represented by multiple overlapping sources in a real world environment [18].

In this paper, an NMF-based adaptive noise sensing and reduction method is proposed to improve the performance of an ASR system under the mismatch in a noise type between the noise basis training and estimation. The proposed method adaptively updates a priori noise basis matrix of the NMF on the fly by estimating the noise signal prior to the actual speech signal. Next, NMF decomposition is carried out with the adapted noise basis matrix in order to estimate clean speech and background noise from noisy speech. Finally, the estimated clean speech signal is applied to a front-end of an ASR system in order to improve the performance of ASR under various noise types.

The rest of this paper is organized as follows. Following this introduction, Section 2 briefly reviews a conventional NMF-based noise reduction method. Section 3 proposes an NMF-based noise sensing and reduction method. Section 4 evaluates the performance of the proposed method and compares it with those of conventional methods. Finally, the paper is concluded in Section 5.

2. Conventional NMF-Based Noise Reduction Method

Figure 1 shows the procedure of a conventional NMF-based noise reduction method applied for an ASR. As shown in the figure, noisy speech is captured by a microphone; then a block of speech signal, called a speech frame, is transformed into a frequency domain by applying a short-time Fourier transform (STFT). Next, an NMF technique is applied to estimate the clean speech spectrum by reducing the noise spectrum. Consequently, the estimated clean spectrum is transformed back into the time domain by applying an inverse STFT. Finally, an ASR system, which is typically based on hidden Markov models [22], is constructed from the feature parameters extracted from this estimated clean speech.

Figure 1

Procedure of a conventional NMF-based noise reduction method.

As mentioned in Figure 1, an NMF-based noise reduction method attempts to decompose a noisy speech signal into separate speech and noise signals by exploring the sparseness of the noisy speech [18]. To explain how to estimate speech and noise with NMF, noisy speech at the ith speech frame, $y_i(n)$ , is represented as

$$\begin{array}{cc}& y_i\left(n\right)=s_i\left(n\right)+d_i\left(n\right),\end{array}$$

(1)

where $s_i(n)$ and $d_i(n)$ are clean speech and additive noise at the ith speech frame, respectively, and $d_i(n)$ is assumed to be uncorrelated with $s_i(n)$ . By applying an STFT to (1), $y_i(n)$ can be represented as the spectral components, as

$$\begin{array}{cc}& Y_i\left(k\right)=S_i\left(k\right)+D_i\left(k\right)\hspace{1em}\text{for}\mathrm{\hspace{0.17em}\hspace{0.17em}}k=\mathrm{0,1},\dots ,K-\mathrm{1},\end{array}$$

(2)

where $Y_i(k),S_i(k)$ , and $D_i(k)$ denote the kth spectral components of $y_i(n),s_i(n)$ , and $d_i(n)$ , respectively. In order to estimate $S_i(k)$ from $Y_i(k)$ , the spectral magnitudes of several speech frames are concatenated together so that $\mathbf{Y}=\mathbf{S}+\mathbf{D}$ is obtained. Note that it is assumed here that $|Y_i(k)|\approx |S_i(k)|+|D_i(k)|$ , because this assumption has provided satisfactory results for NMF-based noise reduction [23, 24]. Thus, the matrices $\mathbf{Y},\mathbf{S}$ , and D are all $K\times N$ matrices, where K and N are the number of frequency bins and the number of concatenated frames, respectively.

In the NMF framework, Y, S, and D are represented as $\mathbf{Y}={\mathbf{B}}_{\mathbf{Y}}{\mathbf{A}}_{\mathbf{Y}}$ , $\mathbf{S}={\mathbf{B}}_{\mathbf{S}}{\mathbf{A}}_{\mathbf{S}}$ , and $\mathbf{D}={\mathbf{B}}_{\mathbf{D}}{\mathbf{A}}_{\mathbf{D}}$ , respectively, while ${\mathbf{B}}_{\mathbf{Y}}$ , ${\mathbf{B}}_{\mathbf{S}}$ , and ${\mathbf{B}}_{\mathbf{D}}$ are the basis matrices of Y, S, and D, respectively, and ${\mathbf{A}}_{\mathbf{Y}}$ , ${\mathbf{A}}_{\mathbf{S}}$ , and ${\mathbf{A}}_{\mathbf{D}}$ are the activation matrices corresponding to ${\mathbf{B}}_{\mathbf{Y}}$ , ${\mathbf{B}}_{\mathbf{S}}$ , and ${\mathbf{B}}_{\mathbf{D}}$ , respectively. If it is assumed that S and D are fully separable from $\mathbf{Y},\mathbf{Y}$ can be rewritten as [23]

$$\begin{array}{l}\mathbf{Y}={\mathbf{B}}_{\mathbf{Y}}{\mathbf{A}}_{\mathbf{Y}}\\ =\left[{\mathbf{B}}_{\mathbf{S}}\hspace{1em}{\mathbf{B}}_{\mathbf{D}}\right]\mathrm{\hspace{0.17em}\hspace{0.17em}}\left[\begin{array}{l}{\mathbf{A}}_{\mathbf{S}}\\ {\mathbf{A}}_{\mathbf{D}}\end{array}\right]={\mathbf{B}}_{\mathbf{S}}{\mathbf{A}}_{\mathbf{S}}+{\mathbf{B}}_{\mathbf{D}}{\mathbf{A}}_{\mathbf{D}}=\mathbf{S}+\mathbf{D},\end{array}$$

(3)

where ${\mathbf{B}}_{\mathbf{Y}}=[{\mathbf{B}}_{\mathbf{S}}\hspace{1em}{\mathbf{B}}_{\mathbf{D}}]$ and ${\mathbf{A}}_{\mathbf{Y}}={[{\mathbf{A}}_{\mathbf{S}}\hspace{1em}{\mathbf{A}}_{\mathbf{D}}]}^T$ . Here, T is a transpose operator. Since $R_S$ and $R_D$ ( $R_Y=R_S+R_D$ ) are the ranks of the basis matrices for S and D, the dimensions of ${\mathbf{B}}_{\mathbf{Y}}$ , ${\mathbf{B}}_{\mathbf{S}}$ , and ${\mathbf{B}}_{\mathbf{D}}$ are $K\times R_Y$ ,  $K\times R_S$ , and $K\times R_D$ , respectively, and the dimensions of ${\mathbf{A}}_{\mathbf{Y}}$ , ${\mathbf{A}}_{\mathbf{S}}$ , and ${\mathbf{A}}_{\mathbf{D}}$ are $R_Y\times N$ , $R_S\times N$ , and $R_D\times N$ , respectively. As described in (3), S and D can be obtained, if ${\mathbf{B}}_{\mathbf{S}}$ , ${\mathbf{B}}_{\mathbf{D}}$ , ${\mathbf{A}}_{\mathbf{S}}$ , and ${\mathbf{A}}_{\mathbf{D}}$ are known a priori. Therefore, most conventional NMF-based noise reduction methods estimate ${\mathbf{B}}_{\tilde{\mathbf{S}}}$ and ${\mathbf{B}}_{\tilde{\mathbf{D}}}$ from previously stored speech and noise databases, $\tilde{\mathbf{S}}$ and $\tilde{\mathbf{D}}$ [14, 15]. For given $\tilde{\mathbf{S}}$ and $\tilde{\mathbf{D}}$ , ${\mathbf{B}}_{\tilde{\mathbf{S}}}$ and ${\mathbf{B}}_{\tilde{\mathbf{D}}}$ can be obtained through a multiplicative update rule [25] as

$$\begin{array}{cc}& {\mathbf{B}}_{\tilde{\mathbf{S}}}^t={\mathbf{B}}_{\tilde{\mathbf{S}}}^{t-\mathrm{1}}\otimes \frac{\left(\tilde{\mathbf{S}}/{\mathbf{B}}_{\tilde{\mathbf{S}}}^{t-\mathrm{1}}{\mathbf{A}}_{\tilde{\mathbf{S}}}^{t-\mathrm{1}}\right){\left({\mathbf{A}}_{\tilde{\mathbf{S}}}^{t-\mathrm{1}}\right)}^T}{\mathbf{1}{\left({\mathbf{A}}_{\tilde{\mathbf{S}}}^{t-\mathrm{1}}\right)}^T},\end{array}$$

(4)

$$\begin{array}{cc}& {\mathbf{A}}_{\tilde{\mathbf{S}}}^t={\mathbf{A}}_{\tilde{\mathbf{S}}}^{t-\mathrm{1}}\otimes \frac{{\left({\mathbf{B}}_{\tilde{\mathbf{S}}}^t\right)}^T\left(\tilde{\mathbf{S}}/{\mathbf{B}}_{\tilde{\mathbf{S}}}^t{\mathbf{A}}_{\tilde{\mathbf{S}}}^{t-\mathrm{1}}\right)}{{\left({\mathbf{B}}_{\tilde{\mathbf{S}}}^t\right)}^T\mathbf{1}},\end{array}$$

(5)

$$\begin{array}{cc}& {\mathbf{B}}_{\tilde{\mathbf{D}}}^t={\mathbf{B}}_{\tilde{\mathbf{D}}}^{t-\mathrm{1}}\otimes \frac{\left(\tilde{\mathbf{D}}/{\mathbf{B}}_{\tilde{\mathbf{D}}}^{t-\mathrm{1}}{\mathbf{A}}_{\tilde{\mathbf{D}}}^{t-\mathrm{1}}\right){\left({\mathbf{A}}_{\tilde{\mathbf{D}}}^{t-\mathrm{1}}\right)}^T}{\mathbf{1}{\left({\mathbf{A}}_{\tilde{\mathbf{D}}}^{t-\mathrm{1}}\right)}^T},\end{array}$$

(6)

$$\begin{array}{cc}& {\mathbf{A}}_{\tilde{\mathbf{D}}}^t={\mathbf{A}}_{\tilde{\mathbf{D}}}^{t-\mathrm{1}}\otimes \frac{{\left({\mathbf{B}}_{\tilde{\mathbf{D}}}^t\right)}^T\left(\tilde{\mathbf{D}}/{\mathbf{B}}_{\tilde{\mathbf{D}}}^t{\mathbf{A}}_{\tilde{\mathbf{D}}}^{t-\mathrm{1}}\right)}{{\left({\mathbf{B}}_{\tilde{\mathbf{D}}}^t\right)}^T\mathbf{1}},\end{array}$$

(7)

where t is an iteration index and both multiplication, ⊗, and division are applied on an element-by-element basis. In addition, $\mathbf{1}$ is a $K\times N$ matrix in which all elements are equal to unity. Note that all elements for ${\mathbf{B}}_{\tilde{\mathbf{S}}}^{\mathrm{0}}$ , ${\mathbf{A}}_{\tilde{\mathbf{S}}}^{\mathrm{0}}$ , ${\mathbf{B}}_{\tilde{\mathbf{D}}}^{\mathrm{0}}$ , and ${\mathbf{A}}_{\tilde{\mathbf{D}}}^{\mathrm{0}}$ can be set initially at random values between 0 and 1.

In the NMF training, ${\mathbf{B}}_{\tilde{\mathbf{S}}}$ is obtained by repeating (4) and (5) until the relative reduction of the NMF objective function according to the iteration arriving at a value below a predefined threshold. In this paper, the Kullback-Leibler (KL) divergence is employed as an NMF objective function [25]. Similarly, ${\mathbf{B}}_{\tilde{\mathbf{D}}}$ is also obtained by repeating (6) and (7), and the estimation process is terminated based on the KL divergence.

As described above, the conventional noise reduction methods are performed using ${\mathbf{B}}_{\mathbf{S}}={\mathbf{B}}_{\tilde{\mathbf{S}}}^{T_S}$ and ${\mathbf{B}}_{\mathbf{D}}={\mathbf{B}}_{\tilde{\mathbf{D}}}^{T_D}$ , where $T_S$ and $T_D$ are the final iterations of the NMF training. Then, the activation matrices, ${\mathbf{A}}_{\mathbf{S}}$ and ${\mathbf{A}}_{\mathbf{D}}$ , are calculated by the multiplicative NMF update rule as

$$\begin{array}{l}\left[\begin{array}{l}{\mathbf{A}}_{\mathbf{S}}^t\\ {\mathbf{A}}_{\mathbf{D}}^t\end{array}\right]\\ \hspace{1em}=\left[\begin{array}{l}{\mathbf{A}}_{\mathbf{S}}^{t-\mathrm{1}}\\ {\mathbf{A}}_{\mathbf{D}}^{t-\mathrm{1}}\end{array}\right]\otimes \frac{{\left[{\mathbf{B}}_{\mathbf{S}}\hspace{1em}{\mathbf{B}}_{\mathbf{D}}\right]}^T\left(\mathbf{Y}/\left[{\mathbf{B}}_{\mathbf{S}}\hspace{1em}{\mathbf{B}}_{\mathbf{D}}\right]{\left[{\mathbf{A}}_{\mathbf{S}}^{t-\mathrm{1}}\hspace{1em}{\mathbf{A}}_{\mathbf{D}}^{t-\mathrm{1}}\right]}^T\right)}{{\left[{\mathbf{B}}_{\mathbf{S}}\hspace{1em}{\mathbf{B}}_{\mathbf{D}}\right]}^T\mathbf{1}},\end{array}$$

(8)

where all elements for ${\mathbf{A}}_{\mathbf{S}}^{\mathrm{0}}$ and ${\mathbf{A}}_{\mathbf{D}}^{\mathrm{0}}$ are also set as random values between 0 and 1. Similar to the NMF training, (8) is repeated until an NMF objective function converges. Subsequently, ${\mathbf{S}}^{\prime }$ is set to ${\mathbf{B}}_{\mathbf{S}}{\mathbf{A}}_{\mathbf{S}}^{T_A}$ , where $T_A$ is the last iteration of (8). Finally, an estimate of clean speech, $s_i^{\prime }(n)$ , is obtained by applying an inverse STFT to ${\mathbf{S}}^{\prime }$ , and it is fed to the front-end of an ASR system.

However, the main drawback of the conventional NMF-based noise reduction methods is that the noise reduction performance is not reliable when there is a mismatch in noise types between the noise basis training and estimation using NMF. In other words, the basis matrices, ${\mathbf{B}}_{\tilde{\mathbf{D}}}^{T_D}$ , are inappropriate for the ASR test. To overcome this problem, ${\mathbf{B}}_{\tilde{\mathbf{D}}}^{T_D}$ should be updated during the ASR test.

3. Proposed NMF-Based Adaptive Noise Sensing and Reduction

In this section, an NMF-based adaptive noise sensing and reduction method is proposed to mitigate the degradation of noise reduction when there is a mismatch in noise types between noise basis training and estimation using NMF. Figure 2 shows the procedure of the proposed NMF-based adaptive noise sensing and reduction method. As shown in the figure, the procedure is divided into three different processing stages: a priori NMF basis modeling, NMF-based adaptive noise sensing, and noise reduction. The first processing stage of the proposed method is the same as that of the conventional method described in Section 2. In other words, clean speech signals and noise signals are separately applied to the NMF training in order to obtain the a priori basis matrices. In the second processing stage, the adaptive noise sensing is performed to decompose the noisy input spectrum into speech and noise spectrum using a priori speech basis matrix estimated by the first processing stage. That is, the noise basis and activation matrices are obtained by adapting a priori noise basis from the instantaneous noise frames of the noisy input signal. Finally, the third processing stage of the proposed method estimates the noise-reduced speech signal by constructing a Wiener filter [8] using the adaptively estimated noise spectrum. The following subsections describe a priori NMF basis acquisition, NMF-based adaptive noise sensing, and noise reduction in detail.

Figure 2

Procedure of the proposed NMF-based adaptive noise sensing and reduction method.

3.1. Modeling of A Priori NMF Basis Matrix

This subsection describes how to obtain the NMF basis matrices of speech and noise signals. As mentioned in Section 2, for given speech and noise database, $\tilde{\mathbf{S}}$ and $\tilde{\mathit{D}}$ , the basis and activation matrices for speech and noise, ${\mathbf{B}}_{\tilde{\mathbf{S}}}$ , ${\mathbf{A}}_{\tilde{\mathbf{S}}}$ , ${\mathbf{B}}_{\tilde{\mathbf{D}}}$ , and ${\mathbf{A}}_{\tilde{\mathbf{D}}}$ , are obtained by iterating equations (4)–(7). Notice that the dimensions of ${\mathbf{B}}_{\tilde{\mathbf{S}}}$ , ${\mathbf{A}}_{\tilde{\mathbf{S}}}$ , ${\mathbf{B}}_{\tilde{\mathbf{D}}}$ , and ${\mathbf{A}}_{\tilde{\mathbf{D}}}$ are $K\times R_S$ , $R_S\times N$ , $K\times R_D$ , and $R_D\times N$ , respectively, where $R_S$ and $R_D$ are the number of bases for $\tilde{\mathit{S}}$ and $\tilde{\mathit{D}}$ . In this paper, K, N, $R_S$ , and $R_D$ are set to 257, 2000, 80, and 40, respectively. In particular, as a termination condition for the NMF iteration, the divergence cost function [25] for (4) and (5) is defined as

$$\begin{array}{l}\text{Div}\left(\tilde{\mathit{S}};{\mathit{A}}_{\tilde{\mathit{S}}}^t,{\mathit{B}}_{\tilde{\mathit{S}}}^t\right)\\ \hspace{1em}=\sum _{K,N}\left|\left(\tilde{\mathit{S}}\otimes \log \left(\frac{\tilde{\mathit{S}}}{\left({\mathit{B}}_{\tilde{\mathit{S}}}^t{\mathit{A}}_{\tilde{\mathit{S}}}^t\right)}\right)\right)-\left(\tilde{\mathit{S}}-{\mathit{B}}_{\tilde{\mathit{S}}}^t{\mathit{A}}_{\tilde{\mathit{S}}}^t\right)\right|,\end{array}$$

(9)

where the summation notation means adding all the elements of a matrix together. Accordingly, if $|\text{Div}(\tilde{\mathbf{S}};{\mathbf{A}}_{\tilde{\mathbf{S}}}^t,{\mathbf{B}}_{\tilde{\mathbf{S}}}^t)-\text{Div}(\tilde{\mathbf{S}};{\mathbf{A}}_{\tilde{\mathbf{S}}}^{t-\mathrm{1}},{\mathbf{B}}_{\tilde{\mathbf{S}}}^{t-\mathrm{1}})|/\text{Div}(\tilde{\mathbf{S}};{\mathbf{A}}_{\tilde{\mathbf{S}}}^{t-\mathrm{1}},{\mathbf{B}}_{\tilde{\mathbf{S}}}^{t-\mathrm{1}})<\theta$ , then ${\mathbf{B}}_{\tilde{\mathbf{S}}}={\mathbf{B}}_{\tilde{\mathbf{S}}}^t$ and ${\mathbf{A}}_{\tilde{\mathbf{S}}}={\mathbf{A}}_{\tilde{\mathbf{S}}}^t$ , where, in this paper, θ is set to 0.001 from the preliminary experiments at the training stage. Similarly, ${\mathbf{B}}_{\tilde{\mathbf{D}}}={\mathbf{B}}_{\tilde{\mathbf{D}}}^t$ and ${\mathbf{A}}_{\tilde{\mathbf{D}}}={\mathbf{A}}_{\tilde{\mathbf{D}}}^t$ are obtained from (6) and (7), if the change in divergence falls within a predefined threshold, such as $|\text{Div}(\tilde{\mathbf{D}};{\mathbf{A}}_{\tilde{\mathbf{S}}}^t,{\mathbf{B}}_{\tilde{\mathbf{S}}}^t)-\text{Div}(\tilde{\mathbf{D}};{\mathbf{A}}_{\tilde{\mathbf{S}}}^{t-\mathrm{1}},{\mathbf{B}}_{\tilde{\mathbf{S}}}^{t-\mathrm{1}})|/\text{Div}(\tilde{\mathbf{D}};{\mathbf{A}}_{\tilde{\mathbf{S}}}^{t-\mathrm{1}},{\mathbf{B}}_{\tilde{\mathbf{S}}}^{t-\mathrm{1}})<\theta$ , where θ is also set to 0.001. Finally, ${\mathbf{B}}_{\tilde{\mathbf{S}}}$ and ${\mathbf{B}}_{\tilde{\mathbf{D}}}$ are applied to the NMF-based adaptive noise sensing stage, which will be explained in the next subsection.

3.2. NMF-Based Adaptive Noise Sensing

This subsection describes how the noise spectrum is adapted into the NMF framework. First, noise frames are detected from noisy input speech. Then, the detected noise frames are concatenated to construct a noise matrix, ${\mathbf{D}}^{\prime }$ , which is used to adapt ${\mathbf{B}}_{\tilde{\mathbf{D}}}$ , which is estimated a priori as described in Section 3.1. Specifically, the ratio of speech and noise magnitudes for the kth frequency bin at the ith frame, $R_i(k)$ , are first calculated as

$$\begin{array}{cc}& R_i\left(k\right)=\frac{\left|{\tilde{S}}_i\left(k\right)\right|}{\left|{\tilde{D}}_i\left(k\right)\right|},\end{array}$$

(10)

where $|{\tilde{S}}_i(k)|$ and $|{\tilde{D}}_i(k)|$ are the clean and noise spectral components estimated from $\tilde{\mathbf{S}}={\mathbf{B}}_{\tilde{\mathbf{S}}}{\mathbf{A}}_{\mathbf{S}}$ and $\tilde{\mathbf{D}}={\mathbf{B}}_{\tilde{\mathbf{D}}}{\mathbf{A}}_{\mathbf{D}}$ , respectively. Then, a speech mask at the ith frame, $m_s(i)$ , is calculated as

$$\begin{array}{cc}& m_s\left(i\right)=\frac{\mathrm{1}}{K}{{\displaystyle \sum }}_{k=\mathrm{0}}^{K-\mathrm{1}}{R}_i\left(k\right).\end{array}$$

(11)

Using (11), a set of noise frames, ${\mathbf{I}}_{\mathbf{D}}$ , is selected which satisfies the equation of

$$\begin{array}{cc}& {\mathbf{I}}_{\mathbf{D}}=\left\{i\hspace{0.17em}|\hspace{0.17em}{m}_s\left(i\right)\le \eta \right\},\end{array}$$

(12)

where η is a threshold for detecting noise frames. In this paper, η is determined by considering the mean and variance of $m_s(i)$ such that

$$\begin{array}{cc}& \eta ={\mu }_m+\kappa {\sigma }_m,\end{array}$$

(13)

where ${\mu }_m=(\mathrm{1}/I){\sum }_{i=\mathrm{1}}^I{m}_s(i)$ and ${\sigma }_m^{\mathrm{2}}=(\mathrm{1}/I){\sum }_{i=\mathrm{1}}^I(m_s(i)-{\mu }_m{)}^{\mathrm{2}}$ . For the mean and variance calculation, the first I frames of noisy speech are assumed to be noise frames and $I=\mathrm{20}$ in this paper. In addition, κ in (13) is set so that approximately 80% of the initial I frames are included in ${\mathbf{I}}_{\mathbf{D}}$ . As a result, a noise binary mask for the kth frequency bin at the ith frame, $M_i(k)$ , can be defined as

$$\begin{array}{cc}& M_i\left(k\right)=\{\begin{array}{cc}\mathrm{1},\hfill & \text{if}\hspace{0.17em}\hspace{0.17em}i\in {\mathbf{I}}_{\mathbf{D}}\hfill \\ \mathrm{0},\hfill & \text{otherwise}\hfill \end{array}\end{array}$$

(14)

and a noise matrix is estimated as ${\mathit{D}}^{\prime }=\mathit{M}\otimes \mathit{Y}$ , where M is a $(K\times N)$ noise mask matrix constructed by (14).

Next, ${\mathbf{B}}_{\widehat{\mathbf{D}}}$ is adapted using ${\mathbf{D}}^{\prime }$ by the following iterative equations of

$$\begin{array}{cc}& {\mathbf{B}}_{\widehat{\mathbf{D}}}^t={\mathbf{B}}_{\widehat{\mathbf{D}}}^{t-\mathrm{1}}\otimes \frac{\left({\mathbf{D}}^{\prime }/{\mathbf{B}}_{\widehat{\mathbf{D}}}^{t-\mathrm{1}}{\mathbf{A}}_{\widehat{\mathbf{D}}}^{t-\mathrm{1}}\right){\left({\mathbf{A}}_{\widehat{\mathbf{D}}}^{t-\mathrm{1}}\right)}^T}{\mathbf{1}{\left({\mathbf{A}}_{\widehat{\mathbf{D}}}^{t-\mathrm{1}}\right)}^T},\\ {\mathbf{A}}_{\widehat{\mathbf{D}}}^t={\mathbf{A}}_{\widehat{\mathbf{D}}}^{t-\mathrm{1}}\otimes \frac{{\left({\mathbf{B}}_{\widehat{\mathbf{D}}}^t\right)}^T\left({\mathbf{D}}^{\prime }/{\mathbf{B}}_{\widehat{\mathbf{D}}}^t{\mathbf{A}}_{\widehat{\mathbf{D}}}^{t-\mathrm{1}}\right)}{{\left({\mathbf{B}}_{\widehat{\mathbf{D}}}^t\right)}^T\mathbf{1}},\end{array}$$

(15)

where ${\mathbf{B}}_{\widehat{\mathbf{D}}}^t$ and ${\mathbf{A}}_{\widehat{\mathbf{D}}}^t$ are the adapted basis and activation matrices of noise at the tth iteration. As an initial condition for (15), all elements of ${\mathbf{A}}_{\widehat{\mathbf{D}}}^{\mathrm{0}}$ can be set as random values between 0 and 1, whereas ${\mathbf{B}}_{\widehat{\mathbf{D}}}^{\mathrm{0}}={\mathbf{B}}_{\tilde{\mathbf{D}}}$ . Similar to the NMF training described in (4)–(7), the procedure of (15) is terminated if the condition $|\text{Div}(\widehat{\mathbf{D}};{\mathbf{A}}_{\widehat{\mathbf{D}}}^t,{\mathbf{B}}_{\widehat{\mathbf{D}}}^t)-\text{Div}(\widehat{\mathbf{D}};{\mathbf{A}}_{\widehat{\mathbf{D}}}^{t-\mathrm{1}},{\mathbf{B}}_{\widehat{\mathbf{D}}}^{t-\mathrm{1}})|/\text{Div}(\widehat{\mathbf{D}};{\mathbf{A}}_{\widehat{\mathbf{D}}}^{t-\mathrm{1}},{\mathbf{B}}_{\widehat{\mathbf{D}}}^{t-\mathrm{1}})<{\theta }_{\text{Adapt}}$ is satisfied. It should be noted that the number of iterations for the noise basis adaptation should be smaller than that of the NMF training to prevent ${\mathbf{B}}_{\widehat{\mathbf{D}}}$ from representing only the basis of ${\mathbf{D}}^{\prime }$ . For this reason, ${\theta }_{\text{Adapt}}$ is set to 0.01, which is 10 times greater than θ used in Section 3.1. Consequently, ${\mathbf{B}}_{\widehat{\mathbf{D}}}={\mathbf{B}}_{\widehat{\mathbf{D}}}^t$ and ${\mathbf{A}}_{\widehat{\mathbf{D}}}={\mathbf{A}}_{\widehat{\mathbf{D}}}^t$ are obtained when the procedure terminates at the tth iteration.

As a final processing step for the adaptation, NMF decomposition is performed in order to calculate ${\mathbf{A}}_{\mathbf{S}}$ and ${\mathbf{A}}_{\mathbf{D}}$ . To this end, a multiplicative update rule with ${\mathbf{B}}_{\tilde{\mathbf{S}}}$ and ${\mathbf{B}}_{\widehat{\mathbf{D}}}$ is applied, where ${\mathbf{B}}_{\tilde{\mathbf{S}}}$ is the basis matrix obtained in Section 3.1. That is, the NMF decomposition also iterates the following equation

$$\begin{array}{l}{\mathbf{B}}_{\tilde{\mathbf{D}}}^{T_D}\left[\begin{array}{l}{\mathbf{A}}_{\mathbf{S}}^t\\ {\mathbf{A}}_{\mathbf{D}}^t\end{array}\right]\\ \hspace{1em}=\left[\begin{array}{l}{\mathbf{A}}_{\mathbf{S}}^{t-\mathrm{1}}\\ {\mathbf{A}}_{\mathbf{D}}^{t-\mathrm{1}}\end{array}\right]\otimes \frac{{\left[{\mathbf{B}}_{\tilde{\mathbf{S}}}\hspace{1em}{\mathbf{B}}_{\widehat{\mathbf{D}}}\right]}^T\left(\mathbf{Y}/\left[{\mathbf{B}}_{\tilde{\mathbf{S}}}\hspace{1em}{\mathbf{B}}_{\widehat{\mathbf{D}}}\right]{\left[{\mathbf{A}}_{\mathbf{S}}^{t-\mathrm{1}}\hspace{1em}{\mathbf{A}}_{\mathbf{D}}^{t-\mathrm{1}}\right]}^T\right)}{{\left[{\mathbf{B}}_{\tilde{\mathbf{S}}}\hspace{1em}{\mathbf{B}}_{\widehat{\mathbf{D}}}\right]}^T\mathbf{1}},\end{array}$$

(16)

where all elements in ${\mathbf{A}}_{\mathbf{S}}^{\mathrm{0}}$ and ${\mathbf{A}}_{\mathbf{D}}^{\mathrm{0}}$ , are set as random values between 0 and 1. The termination condition is also defined as $\text{Div}(\mathbf{Y};[{\mathbf{A}}_{\mathbf{S}}^{t-\mathrm{1}}\hspace{1em}{\mathbf{A}}_{\mathbf{D}}^{t-\mathrm{1}}{]}^T,[{\mathbf{B}}_{\tilde{\mathbf{S}}}\hspace{1em}{\mathbf{B}}_{\widehat{\mathbf{D}}}])$ . Thus, ${\mathbf{A}}_{\mathbf{S}}$ and ${\mathbf{A}}_{\mathbf{D}}$ are set to ${\mathbf{A}}_{\mathbf{S}}^t$ and ${\mathbf{A}}_{\mathbf{D}}^t$ , respectively, when the procedure of (16) is terminated at the tth iteration. Finally, $\widehat{\mathbf{S}}$ and $\widehat{\mathbf{D}}$ are obtained by $\widehat{\mathbf{S}}={\mathbf{B}}_{\tilde{\mathbf{S}}}{\mathbf{A}}_{\mathbf{S}}$ and $\widehat{\mathbf{D}}={\mathbf{B}}_{\widehat{\mathbf{D}}}{\mathbf{A}}_{\mathbf{D}}$ , respectively, and they are used for noise reduction, which will be explained in the next subsection.

3.3. Noise Reduction

This subsection describes how to reduce noise from noisy input speech using the adapted noise basis of NMF, which is the third processing stage of Figure 2. First, a $(K\times N)$ noise attenuation gain, G, is calculated as

$$\begin{array}{cc}& \mathbf{G}=\frac{\widehat{\mathbf{S}}}{\widehat{\mathbf{S}}+\mathrm{\Delta }\otimes \widehat{\mathbf{D}}},\end{array}$$

(17)

where $\mathrm{\Delta }$ is a $(K\times N)$ noise reduction control matrix with all elements equal to a constant, δ, in order to control the degree of noise reduction by scaling $\widehat{\mathbf{D}}$ . In this paper, δ is set to 3 since this value of δ provides the best noise reduction performance. Next, each column of G is applied as a transfer function of the Wiener filter to each ith frame of the noisy input speech, $y_i(n)$ , resulting in an estimation of clean speech $s_i^{\prime }(n)$ [8].

4. Performance Evaluation

The performance of the proposed method was first evaluated by measuring the source to distortion ratio (SDR), source to interferences ratio (SIR), and source to artifacts ratio (SAR) [26]. Next, the average word error rate (WER) of an ASR system employing the proposed method was measured. Finally, the performance of the proposed method was compared with those of the two-stage mel-warped Wiener filter method (Mel-WF) [8] and the NMF-based noise reduction method without noise basis adaptation (NMF-Conv) [14].

For the evaluation, 10 males and 10 females spoke 20 sentences each, resulting in 400 sentences. This recording was performed in a quiet room without any reverberation. Next, each sentence was mixed with four different kinds of background noise recorded at bus stops, restaurants, subways, and a living room with a TV on, where signal-to-noise ratio (SNR) was changed from 0 to 20 dB with a step of 5 dB. The bus stop, restaurant, and subway noises were used to simulate high stationary noise environments, while the living room noise was used in order to simulate a high nonstationary noisy environment in which a person was speaking while watching different genres of TV programs such as drama, news, sports, and movies. It should be noted that the restaurant and living room noise signals were recorded in a nonreverberant room. The speech and noise signals used in the evaluation were sampled at 16 kHz with a 16-bit resolution. A priori basis matrices for the evaluation were prepared as follows. First, a priori basis matrix for speech, ${\mathbf{B}}_{\tilde{\mathbf{S}}}$ , was trained for each individual speaker with a 20-second long clean sentence. Next, a priori basis matrix for noise, ${\mathbf{B}}_{\tilde{\mathbf{D}}}$ , was trained with 60 seconds of cafeteria noise, where the cafeteria noise was different from other four types of background noise used for the performance evaluation.

4.1. Noise Reduction Performance

In this subsection, the noise reduction performance of the proposed method was evaluated under both nonstationary and stationary noise conditions by measuring the SDR, SIR, and SAR. As shown in (1), a noisy speech signal was composed of clean speech and noise as $y(n)=s(n)+d(n)$ , and the estimates of $s(n)$ and $s^{\prime }(n)$ were obtained by using the proposed method described in Section 3. Then, the true clean signal and its estimate were related by $s^{\prime }(n)=s(n)+e_{\text{interf}}(n)+e_{\text{noise}}(n)+e_{\text{artif}}(n)$ , where $e_{\text{interf}}(n),e_{\text{noise}}(n)$ , and $e_{\text{artif}}(n)$ were the errors associated with the interference, noise, and artifacts, respectively, and they were obtained through least-square projection [26]. By using those errors, SDR, SIR, and SAR were defined as [26]

$$\begin{array}{cc}& \text{SDR}=\mathrm{10}{\log }_{\mathrm{10}}\frac{{\parallel s\left(n\right)\parallel }^{\mathrm{2}}}{{\parallel e_{\text{interf}}\left(n\right)+e_{\text{noise}}\left(n\right)+e_{\text{artif}}\left(n\right)\parallel }^{\mathrm{2}}},\\ \text{SIR}=\mathrm{10}{\log }_{\mathrm{10}}\frac{{\parallel s\left(n\right)\parallel }^{\mathrm{2}}}{{\parallel e_{\text{interf}}\left(n\right)\parallel }^{\mathrm{2}}},\\ \text{SAR}=\mathrm{10}{\log }_{\mathrm{10}}\frac{{\parallel s\left(n\right)+e_{\text{interf}}\left(n\right)+e_{\text{noise}}\left(n\right)\parallel }^{\mathrm{2}}}{{\parallel e_{\text{artif}}\left(n\right)\parallel }^{\mathrm{2}}},\end{array}$$

(18)

where $\parallel ·\parallel$ is the norm operator.

First, Table 1 compares the SDRs, SIRs, and SARs of the proposed method and those of the conventional methods under a nonstationary noise condition such as the living room condition. As shown in the table, the proposed method significantly increased the average SDR, SIR, and SAR values, compared to both the Mel-WF and the NMF-Conv. In particular, the proposed method achieved a dramatically higher average SIR than Mel-WF and NMF-Conv, by 15.01 dB and 8.60 dB, respectively, under the living room noise condition. This implies that the proposed method could provide a speech signal with significantly lower interference than the conventional methods under the nonstationary noise condition.

Table 1

Comparison of the average SDRs, SIRs, and SARs (in dB) of the proposed method and the conventional methods under a living room noise condition.

SNR (dB)	Living Room
	Mel-WF			NMF-Conv			Proposed
	SDR	SIR	SAR	SDR	SIR	SAR	SDR	SIR	SAR
20	19.76	23.30	23.48	21.87	26.74	23.78	24.50	32.22	26.18
15	16.14	18.43	21.53	19.66	23.96	21.94	22.91	30.83	25.13
10	11.99	13.41	19.25	16.29	19.76	19.27	20.67	28.80	21.80
5	7.32	8.22	16.67	13.04	16.13	16.54	17.46	26.29	18.83
0	2.24	2.86	14.12	9.04	11.63	13.39	14.22	23.10	15.15
Average	11.49	13.24	19.01	15.98	19.64	18.98	19.83	28.25	21.15

The performance evaluation was then repeated under three different stationary noise conditions such as bus stop, restaurant, and subway noises. Table 2 shows the SDRs, SIRs, and SARs of the noise-reduced signals processed by the proposed and conventional methods under the stationary noise conditions. Similar to the results under the living room noise condition, the proposed method achieved a substantially higher average of SDR, SIR, and SAR than either the Mel-WF or the NMF-Conv under all stationary noise conditions. It could be concluded that the NMF method employing the proposed noise basis adaptation method performed noise reduction more effectively than the conventional methods under both the stationary and nonstationary noise conditions.

Table 2

Comparison of the average SDRs, SIRs, and SARs (in dB) between the proposed method and the conventional methods under stationary noise conditions such as (a) bus stop, (b) restaurant, and (c) subway noise condition.

SNR (dB)	Mel-WF			NMF-Conv			Proposed
	SDR	SIR	SAR	SDR	SIR	SAR	SDR	SIR	SAR
	(a) Bus Stop
20	18.20	24.34	19.56	19.01	22.41	22.81	20.36	25.36	25.52
15	15.04	19.80	17.04	15.85	17.85	20.36	16.41	21.93	20.77
10	11.54	15.01	14.54	11.46	12.76	17.69	13.86	18.11	19.10
5	7.52	9.84	12.14	6.78	7.81	14.39	9.90	13.77	16.36
0	2.74	4.14	10.18	1.66	2.55	11.09	5.56	9.15	11.18
Average	11.01	14.63	14.69	10.95	12.68	17.27	13.66	17.66	19.84
	(b) Restaurant
20	18.77	24.67	20.28	19.55	23.08	22.23	20.82	27.42	24.27
15	15.52	20.05	17.74	16.40	18.88	20.24	17.87	24.14	20.89
10	11.89	15.06	15.24	12.72	14.50	17.79	14.61	20.39	18.67
5	7.56	9.59	12.67	8.11	9.54	14.38	11.38	16.12	16.28
0	2.27	3.56	10.09	3.32	4.49	11.22	6.56	11.45	11.50
Average	11.21	14.59	15.21	12.02	14.10	17.17	14.62	19.90	19.22
	(c) Subway
20	18.55	24.20	20.06	19.81	22.47	23.42	20.41	25.73	25.25
15	15.24	19.65	17.39	15.96	17.98	20.51	17.33	22.88	22.01
10	11.64	14.91	14.76	11.92	13.37	17.80	14.97	19.35	19.86
5	7.61	9.86	12.27	7.26	8.40	14.44	10.78	15.41	16.36
0	2.94	4.37	10.23	2.18	3.08	11.50	7.05	11.22	12.34
Average	11.20	14.60	14.94	11.42	13.06	17.53	14.49	18.92	19.28

Next, the spectrograms obtained by the proposed method were compared with those by the conventional methods. Figure 3 shows the spectrograms of the noise signals, noisy speech signals at 5 dB SNR, and the estimated noise signals obtained by different noise reduction methods under four different background noise conditions. It was shown by pairwise comparison between Figures 3(e)–3(h) and Figures 3(m)–3(p) that the noise reduction performance of the proposed method was comparable to that of the Mel-WF under stationary noise conditions including bus stop and restaurant noise. On the other hand, the proposed method successfully reduced nonstationary noise under the living room noise condition, whereas the Mel-WF failed to handle the nonstationary noise. Furthermore, it was demonstrated by comparing Figures 3(i)–3(l) and Figures 3(m)–3(p) that the proposed method provided more distinctive speech signals than the NMF-Conv under all the noise conditions.

Figure 3

Spectrograms of signals: (a)–(d) noisy speech signals at 5 dB SNR, (e)–(h) noise-reduced speech signals from Mel-WF, (i)–(l) those of NMF-Conv, and (m)–(p) those by the proposed method under four different noise conditions.

4.2. ASR Performance

To evaluate the recognition performance of the proposed noise reduction method in an ASR system, a hidden Markov model (HMM)-based speech recognition system was constructed. To this end, acoustic models based on three-state left-to-right HMMs were first built from 170,000 phonetically balanced words, which were recorded in quiet rooms by 1,800 speakers. Every recorded speech signal was also sampled at 16 kHz at a 16-bit resolution. As a speech recognition feature, 12 mel-frequency cepstral coefficients (MFCCs) with logarithmic energy were extracted and their delta and acceleration coefficients were concatenated, resulting in a 39-dimensional feature vector [27].

Table 3 compares average WERs of an ASR system employing the proposed method as a front-end with those of ASR systems employing the conventional methods under the nonstationary noise condition. As shown in the table, the proposed method significantly reduced average WER than the conventional methods. Specifically, the proposed method relatively reduced average WER by 65.22% and 24.21% compared to the Mel-WF and the NMF-Conv, respectively.

Table 3

Comparison of average word error rates (WERs) (%) of an ASR system employing the proposed method and the conventional methods under a living room noise condition.

SNR (dB)	Living Room
	Noisy	Mel-WF	NMF-Conv	Proposed
20	24.21	26.19	12.04	10.98
15	33.33	34.54	14.81	11.90
10	49.87	51.07	21.03	17.33
5	74.34	70.76	29.26	21.83
0	94.18	93.21	49.40	33.86
Average	55.19	55.15	25.31	19.18

Second, Table 4 compares average WERs of an ASR system employing the proposed method as a front-end with those of ASR systems employing the conventional methods under stationary noise conditions. As shown in the table, the proposed method relatively reduced average WER under bus stop, restaurant, and subway noise conditions by 0.93%, 11.34%, and 6.56% compared to the Mel-WF and 12.13%, 13.10%, and 11.50% compared to the NMF-Conv, respectively. Consequently, it was concluded that the proposed method provided a better ASR performance than the conventional methods under the stationary and nonstationary noise conditions.

Table 4

Comparison of average word error rates (WERs) (%) of an ASR system employing the proposed method and the conventional methods under stationary noise conditions such as (a) bus stop, (b) restaurant, and (c) subway noise condition.

SNR (dB)	(a) Bus Stop				(b) Restaurant				(c) Subway
	Noisy	Mel-WF	NMF-Conv	Proposed	Noisy	Mel-WF	NMF-Conv	Proposed	Noisy	Mel-WF	NMF-Conv	Proposed
20	16.67	10.19	13.62	10.85	17.99	12.30	12.30	12.30	14.81	11.64	13.89	11.77
15	25.66	14.02	16.53	15.34	34.79	17.72	16.27	13.76	20.77	12.43	14.95	13.36
10	60.32	23.02	24.34	21.96	64.02	26.32	25.66	23.54	51.98	19.58	21.16	21.56
5	90.87	46.03	58.86	45.90	94.18	56.35	57.40	44.05	87.83	46.96	47.49	41.53
0	99.74	90.20	93.52	87.70	99.87	90.08	95.24	86.11	99.21	84.59	87.70	75.66
Average	58.65	36.69	41.37	36.35	62.17	40.55	41.37	35.95	54.92	35.08	37.04	32.78

5. Conclusion

In this paper, an NMF-based noise sensing method has been proposed to reduce stationary and nonstationary noises for speech-based applications over wireless sensor networks. The proposed method adapted the initially estimated noise basis matrix on the fly when the noisy input spectrum was applied to a front-end of a speech-based application. After constructing a Wiener filter using the estimated clean speech and noise spectra in the NMF frame, a clean speech signal was estimated and used for speech recognition. The performance of the proposed method was evaluated by measuring the SDR, SIR, and SAR. In addition, the proposed method was applied to an ASR system and then average WER of the ASR system was evaluated. The performance of the proposed method was also compared with those of conventional methods such as the two-stage mel-warped Wiener filter method and the NMF-based noise reduction method without noise basis adaptation. As a result, it was shown that the proposed method provided better performance in terms of the SDR, SIR, SAR, and WER than the conventional methods under both nonstationary and stationary noise conditions.