---
abstract: 'Multichannel processing is widely used for speech enhancement but several limitations appear when trying to deploy these solutions in the real world. Distributed sensor arrays that consider several devices with a few microphones is a viable solution which allows for exploiting the multiple devices equipped with microphones that we are using in our everyday life. In this context, we propose to extend the distributed adaptive node-specific signal estimation approach to a neural network framework. At each node, a local filtering is performed to send one signal to the other nodes where a mask is estimated by a neural network in order to compute a global multichannel Wiener filter. In an array of two nodes, we show that this additional signal can be leveraged to predict the masks and leads to better speech enhancement performance than when the mask estimation relies only on the local signals.'
bibliography:
- 'strings.bib'
- 'refs.bib'
title: 'DNN-based distributed multichannel mask estimation for speech enhancement in microphone arrays'
---

Speech enhancement, microphone arrays, distributed processing.

Introduction {#sec:intro}
============

Almost all voice-based applications such as mobile communications, hearing aids or human to machine interfaces require a clean version of speech for an optimal use. Single-channel speech enhancement can substantially improve the speech intelligibility and speech recognition of a noisy mixture [@Gerkmann2012; @Weninger2014]. However improvement with a single-channel filter is limited by the distortions introduced during the filtering operation. The distortion can be reduced in multichannel processing which exploits spatial information [@Frost1972; @Vincent2018]. The [@Doclo2002] for example yields the optimal filter in the sense and can be extended to a where the noise reduction is balanced by the speech distortion [@Doclo2007].

Up to a certain point, the effectiveness of these algorithms increases with the number of microphones. More microphones can allow for a wider coverage of the acoustic scene and a more accurate estimation of the statistics of the source signals. In large rooms, or even in flats, this implies the need of huge microphone arrays, which, if they are constrained, can become prohibitively expensive and lacks flexibility. However, in our daily life, with the omnipresence of computers, telephones and tablets, we are surrounded by an increased number of embedded microphones. They can be viewed as unconstrained ad hoc microphone arrays which are promising but also challenging [@Bertrand2015]. A algorithm [@Bertrand2010], where the nodes exchange a single linear combination of their local signals, was proposed for a fully connected microphone array. It was shown to converge to the centralized [@Bertrand2010a]. The constraint of a fully connected array can be lifted with randomized gossiping-based algorithms, where beamformer coefficients are computed in a distributed fashion [@Zeng2015]. Message passing [@Heusdens2012] or diffusion-based [@Oconnor2014] algorithms can increase the rather slow convergence rate of these solutions. Another way to exploit the broad covering of the acoustic field by ad hoc microphone arrays is to gather the microphones into clusters dominated by a single common source which can be estimated more efficiently [@Gergen2018].

All these algorithms require the knowledge of either the or the speech activity to compute the filters and are sensitive to signal mismatches [@Vorobyov2003] or detection errors [@Doclo2007]. Deep learning-based approaches have been proposed to estimate accurately these quantities through the prediction of a mask [@Narayanan2013; @Heymann2016; @Perotin2018b] or of the spectrum of the desired signals [@Nugraha2016]. Although often used in a multichannel context, most of these solutions use single-channel data as input of their . Multichannel information was first taken into account through spatial features [@Jiang2014], but can also be exploited using the magnitude and phase of several microphones as the input of a [@Adavanne2018; @Chakrabarty2019]. This yields better results than single-channel prediction but combining all the sensor signals is not scalable and seems suboptimal because of the redundancy of the data. Coping with the redundancy, Perotin et al. [@Perotin2018a] combined a single estimate of the source signals with the input mixture and used the resulting tensor to train a .

In this paper, we consider a fully connected microphone array with synchronized sensors. This allows for using the -based algorithm which was reported to achieve good speech enhancement performance [@Bertrand2010a]. Following the results shown by Perotin et al. [@Perotin2018a], we take advantage of the paradigm [@Bertrand2010a] by combining at each node one local signal with the estimations of the target signal sent by the other nodes. This uses the multichannel context for the mask estimation but avoids the redundancy brought by the signals of a same node. Additionally, this scheme takes advantage of the internal filter operated in and reduces the costs in terms of bandwidth and computational power compared to a network combining all the sensor signals. The paper is organised as follows. The problem formulation and are described in Section \[sec:problem\_formulation\]. In Section \[sec:tf\_estimation\] we present our solution to estimate the masks. The experimental setup is described in Section \[sec:setup\] and results are discussed in Section \[sec:results\] before we conclude the paper.

Problem formulation {#sec:problem_formulation}
===================

Signal model {#subsec:signal_model}
------------

We consider an additive noise model expressed in the domain as $y(f, t) = s(f, t) + n(f, t)$ where $y(f, t)$ is the recorded mixture at frequency index $f$ and time frame index $t$. The speech target signal is denoted $s$ and the noise signal $n$. For the sake of conciseness, we will drop the time and frequency indexes $f$ and $t$. The signals are captured by $M$ microphones and stacked into a vector $\mathbf{y}~=~[y_{1}, ..., y_{M}]^T$. In the following, regular lowercase letters denote scalars; bold lowercase letters indicate vectors and bold uppercase letters indicate matrices.

Multichannel Wiener filter {#subsec:danse}
--------------------------

The operates in a fully connected microphone array. It aims at estimating the speech component $s_{i}$ of a reference signal at microphone $i$. Without loss of generality, we take the reference microphone as $i=1$ in the remainder of the paper. The $\mathbf{w}$ minimises the cost function expressed as follows: $$\label{eq:mse_cost}
J(\mathbf{w}) = \mathbb{E}\{|s_{1} - \mathbf{w}^H\mathbf{y}|^2\}.$$ $\mathbb{E}\{\cdot\}$ is the expectation operator and $\cdot^H$ denotes the Hermitian transpose. The solution to (\[eq:mse\_cost\]) is given by $$\label{eq:mwf_w}
\mathbf{\hat{w}} = \mathbf{R}_{yy}^{-1}\mathbf{R}_{ys}\mathbf{e}_1\,,$$ with $\mathbf{R}_{yy} = \mathbb{E}\{\mathbf{y}\mathbf{y}^H\}$, $\mathbf{R}_{ys} = \mathbb{E}\{\mathbf{y}\mathbf{s}^H\}$ and $\mathbf{e}_1 = [1\; 0 \cdots 0]^T$. Under the assumption that speech and noise are uncorrelated and that the noise is locally stationary, $\mathbf{R}_{ys} = \mathbf{R}_{ss} = \mathbb{E}\{\mathbf{s}\mathbf{s}^H\} = \mathbf{R}_{yy} - \mathbf{R}_{nn}$ where $\mathbf{R}_{nn} = \mathbb{E}\{\mathbf{n}\mathbf{n}^H\}$. Computing these matrices requires the knowledge of noise-only periods and speech-plus-noise periods. This is typically obtained with a [@Doclo2007; @Bertrand2010a].

The provides a trade-off between the noise reduction and the speech distortion [@Doclo2007]. The filter parameters minimise the cost function $$\label{eq:cost_sdw}
J(\mathbf{w}) = \mathbb{E}\{|s_{1} - \mathbf{w}^H\mathbf{s}|^2\} + \mu \mathbb{E}\{|\mathbf{w}^H\mathbf{n}|^2\}\,,$$ with $\mu$ the trade-off parameter. The solution to (\[eq:cost\_sdw\]) is given by $$\label{eq:sdw_w}
\mathbf{\hat{w}} = \big(\mathbf{R}_{ss} + \mu\mathbf{R}_{nn}\big)^{-1}\mathbf{R}_{ss}\mathbf{e}_1.$$ If the desired signal comes from a single source, the speech covariance matrix is theoretically of rank 1. Under this assumption, Serizel et al. [@Serizel2014] proposed a rank-1 approximation of $\mathbf{R}_{ss}$ based on a , delivering a filter that is more robust in low SNR scenarios and provides a stronger noise reduction.

In this section, we briefly describe the algorithm under the assumption that a single target source is present. We consider $M$ microphones spread over $K$ nodes, each node $k$ containing $M_k$ microphones. The signals of one node $k$ are stacked in $\mathbf{y}_k~=~[y_{k,1}, ..., y_{k,M_k}]^T$. As can be seen in (\[eq:mwf\_w\]), the array wide should be computed from all signals of the array, which can result in high bandwidth and computational costs. In , only a single compressed signal $z_j$ is sent from node $j$ to the other nodes. So a node $k$ has $M_k + K -1$ signals, stacked in $\tilde{\mathbf{y}}_k = \left[ \mathbf{y}_k^T,~\mathbf{z}_{-k}^T\right]^T$, where $\mathbf{z}_{-k}$ is a column vector gathering the compressed signals coming from the other nodes ${j\ne k}$. Replacing $\mathbf{y}$ by $\tilde{\mathbf{y}}_k$ and solving (\[eq:cost\_sdw\]) yields the solution to the : $$\label{eq:danse_w}
\mathbf{\tilde{w}}_k = \big(\mathbf{R}_{ss,k} + \mu\mathbf{R}_{nn,k}\big)^{-1}\mathbf{R}_{ss,k}\mathbf{e}_1\,,$$ where $\mathbf{\tilde{w}}_k$, the filter at node $k$, can be decomposed into two filters as $\mathbf{\tilde{w}}_k~=~\left[ \mathbf{w}_{kk}^T,~\mathbf{g}_{k-k}^T\right]$. The first filter $\mathbf{w}_{kk}^T$ is applied on the local signals and $\mathbf{g}_{k-k}$ is applied on the compressed signals sent from the other nodes. The covariance matrices $\mathbf{R}_{ss,k}$ and $\mathbf{R}_{nn,k}$ are computed from the speech and noise components of $\mathbf{\tilde{y}}_k$. The compressed signal $z_{k}$ is computed as $z_{k} =  \mathbf{w}_{kk}^H\mathbf{y}_k$. Bertrand and Moonen proved that this solution converges to the solution with $\mu = 1$, while dividing the bandwidth load by a factor $M_k$ at each node [@Bertrand2010a].

In this paper, we will focus on the batch-mode algorithm where the speech and noise statistics are computed based on the whole signal in order to focus on the interactions between the mask estimated by the and the filters.

Deep neural network based distributed multichannel Wiener filter {#sec:tf_estimation}
================================================================

Heymann et al. predicted masks out of a single signal of the microphone array [@Heymann2016]. Perotin et al. [@Perotin2018a] or Chakrabarty and Habets [@Chakrabarty2019] included several other signals to improve the speech recognition or speech enhancement performance. We propose to extend these scenarios to the multi-node context of . In , at node $k$, a single is used to estimate the source and noise statistics required for both filters $\mathbf{w}_{kk}$ and $\mathbf{w}_{k}$. The first part of our contribution is to replace the by a mask predicted by a . Besides, since the compressed signals ${z}_{k}$ are sent from one node to the others, we also examine the option of exploiting this extra source of information by using it for the mask prediction. The schematic principle of is depicted in Figure \[fig:danse\]. As it can be seen, an initialisation phase is required to compute the initial signal $z_k$. We propose to do this with a first neural network. The second stage of is represented in the greyed box in Figure \[fig:danse\] and expended in Figure \[fig:solution\]. Our second contribution is highlighted with the red arrow. It is to exploit the presence of $\mathbf{z}_{-k}$ at one node to better predict the masks with the . Several iterations are necessary for the filter $\mathbf{w}_{kk}$ to converge to the solution (\[eq:sdw\_w\]). In , iterations are done at every time step. As we developed an offline batch-mode algorithm, we stopped the processing after the first iteration. To analyse the effectiveness of combining $\mathbf{z}_{-k}$ with a reference signal to predict the mask, we compare our solution with a single-channel prediction, where the masks required for both initialisation and iteration stages are predicted by a single-channel model seeing only the local signal $y_{k,1}$.

We compare two different architectures for each of these schemes. The first architecture is a bidirectional introduced by Heymann et al. [@Heymann2016]. When additional inputs are used with a , they are stacked over the frequency axis [@Perotin2018a]. Although this might deliver improved performance compared to the single-channel version, stacking it over the frequency axis is not efficient as many connections are used to represent relations between bins that might not be related. That is why we propose a architecture which is more appropriate to process multichannel data. At each node, the compressed signals $\mathbf{z}_{-k}$ and the local reference signal $y_{k,1}$ are considered as separate convolutional channels.

During the training, in order to take into account the spectral shape of the speech, we weight the loss between the predicted mask $\hat{\mathbf{m}}$ and the ground truth mask $\mathbf{m}$ by the frame of the input $\mathbf{y}$, corresponding to the predicted frame. Both models are thus trained to minimise the cost function $$\mathcal{L}(\mathbf{m}, \hat{\mathbf{m}}) = E\{|(\mathbf{m} - \hat{\mathbf{m}})\cdot\mathbf{y}|^2\},$$ where $E\{\cdot\}$ represents the empirical mean.

Lastly, since the filter $\mathbf{w}_{kk}$ is also applied on $\mathbf{z}_{-k}$, we use the of the covariance matrices to compute the of equation (\[eq:sdw\_w\]). Contrary to equation (\[eq:mwf\_w\]), this does not explicitly take the first microphone as a reference. It also assigns higher importance to the compressed signals, which is desirable since they are pre-filtered with potentially higher than the local signals.

![Block diagram of principle. Bold arrows represent vectors, simple ones represent scalars.[]{data-label="fig:danse"}](danse_block.pdf){width="\linewidth"}

![Expansion of the iterated step in Figure \[fig:danse\]. Red parts are the modifications proposed to . Bold arrows represent multichannel signals.[]{data-label="fig:solution"}](solution.pdf){width="\linewidth"}

Experimental setup {#sec:setup}
==================

Dataset {#subsec:data}
-------

Training as well as test data was generated by convolving clean speech and noise signals with simulated , and then by mixing the convolved signals at a specific . The anechoic speech material was taken from the clean subset of LibriSpeech [@Panayotov2015]. The were obtained with the Matlab toolbox *Roomsimove*[^1] simulating shoebox-like rooms.

In the training set, the length of the room was drawn uniformly as $l \in [3, 8]$m, the width as $w \in [3, 5]$m, the height as $h \in [2, 3]$m. Two nodes of four microphones each recorded the acoustic scene. The distance between the nodes was set to $1$m, the microphones being $10$cm away from the node centre. Each node was at least 1m away from the closest wall. One source of noise and one of speech were placed at $2.5$ m from the array centre. Both sources had an angular distance $\alpha \in [25, 90]^{\circ}$ relative to the array centre. The microphones as well as the sources were at the constant height of $1.5$ m. The was drawn uniformly between $-5$dB and $+15$dB. The noise was white noise modulated in the spectral domain by the long term spectrum of speech. We generated $10,000$ files of 10 seconds each, corresponding to about 25 hours of training material.

The test configuration was the same as the training configuration but with restricted values for some parameters. The length of the room was randomly selected among $l \in \llbracket3, 8\rrbracket$m, the width among $w \in \llbracket3, 5\rrbracket$m, and the height was set to $h = 2.5$m. The angular distance $\alpha$ between the sources was randomly selected in $\alpha = \{25, 45, 90\}^{\circ}$. The noise was a random part of the third CHiME challenge dataset [@chime] in the cafeteria or pedestrian environment. We generated $1,000$ files representing about 2 hours of test material.

Setup {#subsec:setup}
-----

All the data was sampled at 16 kHz. The was computed with an FFT-length of 512 samples (32 ms), 50% overlap and a Hanning window.

Our model was composed of three convolutional layers with 32, 64 and 64 filters respectively. They all had $3\times3$ kernels, with stride $1\times1$ and ReLU activation functions. Each convolutional layer was followed by a batch normalization over the frequency axis and a maximum pooling layer of size $4\times1$ (along the frequency axis). The recurrent part of the network was a layer with 256 gated recurrent units, and the last layer was a fully connected layer with a sigmoid activation function. The input data of both CRNN and RNN networks was made of sequences of 21 frames and the mask corresponding to the middle frame was predicted. We trained them with the optimizer [@rmsprop].

Results {#sec:results}
=======

We evaluate the speech enhancement performance based on the , and [@Vincent2006] computed with the *mir\_eval*[^2] toolbox. The performance reported corresponds to the mean over the $1,000$ test samples of the objective measures computed at the node with the best input . We also report the 95% confidence interval.

The filter does not explicitly take one sensor signal as the reference signal to minimise the cost function, but a projection of the input signals into the space spanned by the common eigenvectors of the covariance matrices. Because of that, the objective measures computed with respect to the convolved signals did not give results that were coherent with perceptual listening tests performed internally on random samples. Indeed, differences between the enhanced signal and the reference signal are interpreted as artefacts whereas they are due to the decomposition of the input signals into the eigenvalue space of the covariance matrices. Therefore, we compute the objective measures using the dry (source) signals as reference signals. This decreases the because the reverberation is then considered as an artefact but the comparison between methods correlates more with the perceptual listening tests.

We present the objective metrics for the different approaches in Table \[tab:res\_oim\]. In this table, single node filters are referred to as MWF (upper part of the table) and distributed filters as DANSE (lower part of the table). For each filter, the architecture used to obtain the masks is indicated between parenthesis. RNN refers to Heymann’s architecture and CRNN to the network introduced in Section \[subsec:setup\]. The subscript of the network architecture indicates the channels considered at the input. The results obtained with the single-channel models are denoted with “SC”. When the compressed signals $\mathbf{z}_{-k}$ were used as additional input to the to predict the mask of the second filtering stage, models are denoted with “MC”. Additionally, we report the number of trainable parameters of each model in Table \[tab:params\].

Oracle performance {#subsec:oracle_se}
------------------

The gives information about the speech-plus-noise and noise-only periods in a wide-band manner only, whereas a mask gives spectral information that enables a finer estimation of the speech and noise covariance matrices. This additional information is translated into an improvement of the speech enhancement performance with both types of filters (MWF and DANSE). In the following section, we analyse whether this conclusion still holds when the masks are predicted by a neural network.

Performance with predicted masks {#subsec:mc_mp}
--------------------------------

               (dB)                    SAR             SIR            SDR
  ------------------------------ --------------- --------------- --------------
          MWF (oracle )            2.4$\pm$0.3    24.7$\pm$0.3    2.3$\pm$0.3
        MWF (oracle mask)          4.0$\pm$0.3    26.7$\pm$0.3    3.9$\pm$0.3
            MWF (RNN)              3.4$\pm$0.3    25.1$\pm$0.4    3.3$\pm$0.3
            MWF (CRNN)             3.3$\pm$0.3    25.1$\pm$0.4    3.2$\pm$0.3
        DANSE (oracle VAD)        2.6$\pm$ 0.3    25.2$\pm$ 0.3   2.6$\pm$ 0.3
       DANSE (oracle mask)        4.8$\pm$ 0.3    27.6$\pm$ 0.3   4.8$\pm$ 0.3
   DANSE (RNN$_{\mathrm{SC}}$)    $4.0 \pm 0.3$   26.0$\pm$0.4    $4.0\pm0.3$
   DANSE (CRNN$_{\mathrm{SC}}$)    $4.0\pm0.3$    $26.0\pm0.4$    $4.0\pm0.3$
   DANSE (RNN$_{\mathrm{MC}}$)    $4.1\pm 0.3$    $26.1\pm0.4$    $4.0\pm0.3$
   DANSE (CRNN$_{\mathrm{MC}}$)    4.7$\pm$0.3    27.4$\pm$0.3    4.6$\pm$0.3

  : Speech enhancement results in dB with oracle activity detectors and predicted ones.[]{data-label="tab:res_oim"}

First, replacing the oracle by masks brings significant improvement in terms of all objective measures. This confirms the idea that masks are better activity detectors than , even oracle ones. Second, the objective measures corresponding to the output signals of DANSE filters are always better than those of the MWF filters. This confirms the benefit of using the algorithm. Although these differences are not high, increasing the number of nodes and the distance between them might enhance the utility of the distributed method.

From the results in Table \[tab:res\_oim\], there is no clear advantage of using a over using a in the single channel case. Indeed, the objective measures of RNN$_{\mathrm{SC}}$ and CRNN$_{\mathrm{SC}}$ match in all points. In the multichannel case, the performance of the -based approach does not increase. This tends to confirm that the is not able to efficiently exploit multichannel information. Since the delivered good results in the single-channel scenario, this leads to the conclusion that stacking multichannel input on the frequency axis is not appropriate. In addition, as shown in Table \[tab:params\], the number of parameters of the almost doubles when a second signal is used, whereas it barely increases for the . This is due to the convolutional layers of the which can process multichannel data much more efficiently than recurrent layers.

The solution can exploit the multichannel inputs efficiently and the performance increases for all metrics. The biggest improvement is obtained for the . Indeed, one of the main difficulties for the models is to predict noise-only regions, because of people talking in the noise CHiME database. Since the compressed signals are pre-filtered, they contain less noise and they are less ambiguous. This makes it easier for the model to recognize noise-only regions, without degrading its predictions of speech-plus-noise regions.

          Model           Number of parameters
  ---------------------- ----------------------
   RNN$_{\mathrm{SC}}$        $1,717,773 $
   CRNN$_{\mathrm{SC}}$        $911,109$
   RNN$_{\mathrm{MC}}$        $2,244,109$
   CRNN$_{\mathrm{MC}}$        $911,397$

  : Number of trainable parameters of the neural networks.[]{data-label="tab:params"}

Conclusion and future work {#sec:conclusion}
==========================

We introduced an efficient way of estimating masks in a multi-node context. We developed multichannel models combining an estimation of the target signals sent by the other nodes with a local sensor. This proved to better predict masks, which led to higher speech enhancement performance that outperformed the results obtained with an oracle . A was compared to a and the could exploit much better the multichannel information. In addition, the architecture is limited by its number of parameters, especially if the number of nodes had to increase. In such scenarios, the difference between single-channel and multichannel models performance might be even more important but this still has to be explored. To attain performance closer to the oracle ones, several options are possible. First, the rather simple architectures that were used could be replaced by state-of-the art architectures. Besides, given the increase in performance when the target estimation is given, it would also be interesting to additionally give the noise estimation at the input of the models.

\[sec:refs\]

[^1]: homepages.loria.fr/evincent/software/Roomsimove\_1.4.zip

[^2]: https://github.com/craffel/mir\_eval/
