An overview of informed audio source separation
---
paper_title: Probabilistic modeling paradigms for audio source separation

paper_content:

Most sound scenes result from the superposition of several sources, which can be separately perceived and analyzed by human listeners. Source separation aims to provide machine listeners with similar skills by extracting the sounds of individual sources from a given scene. Existing separation systems operate either by emulating the human auditory system or by inferring the parameters of probabilistic sound models. In this chapter, we focus on the latter approach and provide a joint overview of established and recent models, including independent component analysis, local time-frequency models and spectral template-based models. We show that most models are instances of one of the following two general paradigms: linear modeling or variance modeling. We compare the merits of either paradigm and report objective performance figures. We conclude by discussing promising combinations of probabilistic priors and inference algorithms that could form the basis of future state-of-the-art systems.
---
paper_title: Under-Determined Reverberant Audio Source Separation Using a Full-Rank Spatial Covariance Model

paper_content:

This paper addresses the modeling of reverberant recording environments in the context of under-determined convolutive blind source separation. We model the contribution of each source to all mixture channels in the time-frequency domain as a zero-mean Gaussian random variable whose covariance encodes the spatial characteristics of the source. We then consider four specific covariance models, including a full-rank unconstrained model. We derive a family of iterative expectation-maximization (EM) algorithms to estimate the parameters of each model and propose suitable procedures adapted from the state-of-the-art to initialize the parameters and to align the order of the estimated sources across all frequency bins. Experimental results over reverberant synthetic mixtures and live recordings of speech data show the effectiveness of the proposed approach.
---
paper_title: Gamma Markov Random fields for audio source modelling

paper_content:

Audio processing tasks, such as source separation or denoising, require the construction of realistic models that reflect physical properties of audio signals. In this paper, we modelled the variances of time-frequency coefficients of audio signals with Gamma Markov random fields (GMRFs) so that the dependencies between coefficients are captured. There is positive correlation between consecutive variance variables in this model and the strength of this correlation is determined by the coupling hyperparameters. Inference can be carried out using the Gibbs sampler or variational Bayes because the model is conditionally conjugate. However, the optimisation of the hyperparameters is not straightforward because of the intractable normalising constant. In this work, we used this model in denoising and single-channel source separation problems. The hyperparameters of the model are optimised using contrastive divergence and inference is performed using the Gibbs sampler
---
paper_title: Under-Determined Reverberant Audio Source Separation Using a Full-Rank Spatial Covariance Model

paper_content:

This paper addresses the modeling of reverberant recording environments in the context of under-determined convolutive blind source separation. We model the contribution of each source to all mixture channels in the time-frequency domain as a zero-mean Gaussian random variable whose covariance encodes the spatial characteristics of the source. We then consider four specific covariance models, including a full-rank unconstrained model. We derive a family of iterative expectation-maximization (EM) algorithms to estimate the parameters of each model and propose suitable procedures adapted from the state-of-the-art to initialize the parameters and to align the order of the estimated sources across all frequency bins. Experimental results over reverberant synthetic mixtures and live recordings of speech data show the effectiveness of the proposed approach.
---
paper_title: Gaussian Processes for Underdetermined Source Separation

paper_content:

Gaussian process (GP) models are very popular for machine learning and regression and they are widely used to account for spatial or temporal relationships between multivariate random variables. In this paper, we propose a general formulation of underdetermined source separation as a problem involving GP regression. The advantage of the proposed unified view is first to describe the different underdetermined source separation problems as particular cases of a more general framework. Second, it provides a flexible means to include a variety of prior information concerning the sources such as smoothness, local stationarity or periodicity through the use of adequate covariance functions. Third, given the model, it provides an optimal solution in the minimum mean squared error (MMSE) sense to the source separation problem. In order to make the GP models tractable for very large signals, we introduce framing as a GP approximation and we show that computations for regularly sampled and locally stationary GPs can be done very efficiently in the frequency domain. These findings establish a deep connection between GP and nonnegative tensor factorizations (NTF) with the Itakura-Saito distance and lead to effective methods to learn GP hyperparameters for very large and regularly sampled signals.
---
paper_title: Probabilistic modeling paradigms for audio source separation

paper_content:

Most sound scenes result from the superposition of several sources, which can be separately perceived and analyzed by human listeners. Source separation aims to provide machine listeners with similar skills by extracting the sounds of individual sources from a given scene. Existing separation systems operate either by emulating the human auditory system or by inferring the parameters of probabilistic sound models. In this chapter, we focus on the latter approach and provide a joint overview of established and recent models, including independent component analysis, local time-frequency models and spectral template-based models. We show that most models are instances of one of the following two general paradigms: linear modeling or variance modeling. We compare the merits of either paradigm and report objective performance figures. We conclude by discussing promising combinations of probabilistic priors and inference algorithms that could form the basis of future state-of-the-art systems.
---
paper_title: Under-Determined Reverberant Audio Source Separation Using a Full-Rank Spatial Covariance Model

paper_content:

This paper addresses the modeling of reverberant recording environments in the context of under-determined convolutive blind source separation. We model the contribution of each source to all mixture channels in the time-frequency domain as a zero-mean Gaussian random variable whose covariance encodes the spatial characteristics of the source. We then consider four specific covariance models, including a full-rank unconstrained model. We derive a family of iterative expectation-maximization (EM) algorithms to estimate the parameters of each model and propose suitable procedures adapted from the state-of-the-art to initialize the parameters and to align the order of the estimated sources across all frequency bins. Experimental results over reverberant synthetic mixtures and live recordings of speech data show the effectiveness of the proposed approach.
---
paper_title: A Musically Motivated Mid-Level Representation for Pitch Estimation and Musical Audio Source Separation

paper_content:

When designing an audio processing system, the target tasks often influence the choice of a data representation or transformation. Low-level time-frequency representations such as the short-time Fourier transform (STFT) are popular, because they offer a meaningful insight on sound properties for a low computational cost. Conversely, when higher level semantics, such as pitch, timbre or phoneme, are sought after, representations usually tend to enhance their discriminative characteristics, at the expense of their invertibility. They become so-called mid-level representations. In this paper, a source/filter signal model which provides a mid-level representation is proposed. This representation makes the pitch content of the signal as well as some timbre information available, hence keeping as much information from the raw data as possible. This model is successfully used within a main melody extraction system and a lead instrument/accompaniment separation system. Both frameworks obtained top results at several international evaluation campaigns.
---
paper_title: Audio source separation with a single sensor

paper_content:

In this paper, we address the problem of audio source separation with one single sensor, using a statistical model of the sources. The approach is based on a learning step from samples of each source separately, during which we train Gaussian scaled mixture models (GSMM). During the separation step, we derive maximum a posteriori (MAP) and/or posterior mean (PM) estimates of the sources, given the observed audio mixture (Bayesian framework). From the experimental point of view, we test and evaluate the method on real audio examples.
---
paper_title: Gaussian Processes for Underdetermined Source Separation

paper_content:

Gaussian process (GP) models are very popular for machine learning and regression and they are widely used to account for spatial or temporal relationships between multivariate random variables. In this paper, we propose a general formulation of underdetermined source separation as a problem involving GP regression. The advantage of the proposed unified view is first to describe the different underdetermined source separation problems as particular cases of a more general framework. Second, it provides a flexible means to include a variety of prior information concerning the sources such as smoothness, local stationarity or periodicity through the use of adequate covariance functions. Third, given the model, it provides an optimal solution in the minimum mean squared error (MMSE) sense to the source separation problem. In order to make the GP models tractable for very large signals, we introduce framing as a GP approximation and we show that computations for regularly sampled and locally stationary GPs can be done very efficiently in the frequency domain. These findings establish a deep connection between GP and nonnegative tensor factorizations (NTF) with the Itakura-Saito distance and lead to effective methods to learn GP hyperparameters for very large and regularly sampled signals.
---
paper_title: Blind separation of speech mixtures via time-frequency masking

paper_content:

Binary time-frequency masks are powerful tools for the separation of sources from a single mixture. Perfect demixing via binary time-frequency masks is possible provided the time-frequency representations of the sources do not overlap: a condition we call W-disjoint orthogonality. We introduce here the concept of approximate W-disjoint orthogonality and present experimental results demonstrating the level of approximate W-disjoint orthogonality of speech in mixtures of various orders. The results demonstrate that there exist ideal binary time-frequency masks that can separate several speech signals from one mixture. While determining these masks blindly from just one mixture is an open problem, we show that we can approximate the ideal masks in the case where two anechoic mixtures are provided. Motivated by the maximum likelihood mixing parameter estimators, we define a power weighted two-dimensional (2-D) histogram constructed from the ratio of the time-frequency representations of the mixtures that is shown to have one peak for each source with peak location corresponding to the relative attenuation and delay mixing parameters. The histogram is used to create time-frequency masks that partition one of the mixtures into the original sources. Experimental results on speech mixtures verify the technique. Example demixing results can be found online at http://alum.mit.edu/www/rickard/bss.html.
---
paper_title: Oracle estimators for the benchmarking of source separation algorithms

paper_content:

Source separation is a difficult problem for which many algorithms have been proposed. In this article, we define oracle estimators which compute the best performance achievable by different classes of algorithms on a given mixture, in a theoretical evaluation framework where the reference sources are available. We describe explicit oracle estimators for three particular classes of algorithms: multichannel time-invariant filtering, single-channel time-frequency masking and multichannel time-frequency masking. We evaluate their performance on various audio mixtures and study their robustness. We draw several conclusions for their three typical applications, namely providing performance bounds for existing and future blind algorithms, selecting the best class of algorithms for a given mixture and assessing the separation difficulty. In particular, we show that it is worth developing blind time-frequency masking algorithms relaxing the common assumption of a single active source per time-frequency point.
---
paper_title: Under-Determined Reverberant Audio Source Separation Using a Full-Rank Spatial Covariance Model

paper_content:

This paper addresses the modeling of reverberant recording environments in the context of under-determined convolutive blind source separation. We model the contribution of each source to all mixture channels in the time-frequency domain as a zero-mean Gaussian random variable whose covariance encodes the spatial characteristics of the source. We then consider four specific covariance models, including a full-rank unconstrained model. We derive a family of iterative expectation-maximization (EM) algorithms to estimate the parameters of each model and propose suitable procedures adapted from the state-of-the-art to initialize the parameters and to align the order of the estimated sources across all frequency bins. Experimental results over reverberant synthetic mixtures and live recordings of speech data show the effectiveness of the proposed approach.
---
paper_title: Under-Determined Reverberant Audio Source Separation Using a Full-Rank Spatial Covariance Model

paper_content:

This paper addresses the modeling of reverberant recording environments in the context of under-determined convolutive blind source separation. We model the contribution of each source to all mixture channels in the time-frequency domain as a zero-mean Gaussian random variable whose covariance encodes the spatial characteristics of the source. We then consider four specific covariance models, including a full-rank unconstrained model. We derive a family of iterative expectation-maximization (EM) algorithms to estimate the parameters of each model and propose suitable procedures adapted from the state-of-the-art to initialize the parameters and to align the order of the estimated sources across all frequency bins. Experimental results over reverberant synthetic mixtures and live recordings of speech data show the effectiveness of the proposed approach.
---
paper_title: Audio source separation with a single sensor

paper_content:

In this paper, we address the problem of audio source separation with one single sensor, using a statistical model of the sources. The approach is based on a learning step from samples of each source separately, during which we train Gaussian scaled mixture models (GSMM). During the separation step, we derive maximum a posteriori (MAP) and/or posterior mean (PM) estimates of the sources, given the observed audio mixture (Bayesian framework). From the experimental point of view, we test and evaluate the method on real audio examples.
---
paper_title: Probabilistic model for main melody extraction using Constant-Q transform

paper_content:

Dimension reduction techniques such as Nonnegative Tensor Factorization are now classical for both source separation and estimation of multiple fundamental frequencies in audio mixtures. Still, few studies jointly addressed these tasks so far, mainly because separation is often based on the Short Term Fourier Transform (STFT) whereas recent music analysis algorithms are rather based on the Constant-Q Transform (CQT). The CQT is practical for pitch estimation because a pitch shift amounts to a translation of the CQT representation, whereas it produces a scaling of the STFT. Conversely, no simple inversion of the CQT was available until recently, preventing it from being used for source separation. Benefiting from advances both in the inversion of the CQT and in statistical modeling, we show how recent techniques designed for music analysis can also be used for source separation with encouraging results, thus opening the path to many crossovers between separation and analysis.
---
paper_title: Multichannel Nonnegative Matrix Factorization in Convolutive Mixtures for Audio Source Separation

paper_content:

We consider inference in a general data-driven object-based model of multichannel audio data, assumed generated as a possibly underdetermined convolutive mixture of source signals. We work in the short-time Fourier transform (STFT) domain, where convolution is routinely approximated as linear instantaneous mixing in each frequency band. Each source STFT is given a model inspired from nonnegative matrix factorization (NMF) with the Itakura-Saito divergence, which underlies a statistical model of superimposed Gaussian components. We address estimation of the mixing and source parameters using two methods. The first one consists of maximizing the exact joint likelihood of the multichannel data using an expectation-maximization (EM) algorithm. The second method consists of maximizing the sum of individual likelihoods of all channels using a multiplicative update algorithm inspired from NMF methodology. Our decomposition algorithms are applied to stereo audio source separation in various settings, covering blind and supervised separation, music and speech sources, synthetic instantaneous and convolutive mixtures, as well as professionally produced music recordings. Our EM method produces competitive results with respect to state-of-the-art as illustrated on two tasks from the international Signal Separation Evaluation Campaign (SiSEC 2008).
---
paper_title: A Musically Motivated Mid-Level Representation for Pitch Estimation and Musical Audio Source Separation

paper_content:

When designing an audio processing system, the target tasks often influence the choice of a data representation or transformation. Low-level time-frequency representations such as the short-time Fourier transform (STFT) are popular, because they offer a meaningful insight on sound properties for a low computational cost. Conversely, when higher level semantics, such as pitch, timbre or phoneme, are sought after, representations usually tend to enhance their discriminative characteristics, at the expense of their invertibility. They become so-called mid-level representations. In this paper, a source/filter signal model which provides a mid-level representation is proposed. This representation makes the pitch content of the signal as well as some timbre information available, hence keeping as much information from the raw data as possible. This model is successfully used within a main melody extraction system and a lead instrument/accompaniment separation system. Both frameworks obtained top results at several international evaluation campaigns.
---
paper_title: Algorithms for probabilistic latent tensor factorization

paper_content:

We propose a general probabilistic framework for modelling multiway data. Our approach establishes a novel link between graphical representation of probability measures and tensor factorization models that allow us to design arbitrary tensor factorization models while retaining simplicity. Using an expectation-maximization (EM) approach for maximizing the likelihood of the exponential dispersion models (EDM), we obtain iterative update equations for Kullback-Leibler (KL), Euclidian (EU) or Itakura-Saito (IS) costs as special cases. Besides EM, we derive alternative algorithms with multiplicative update rules (MUR) and alternating projections. We also provide algorithms for MAP estimation with conjugate priors. All of the algorithms can be formulated as message passing algorithm on a graph where vertices correspond to indices and cliques represent factors of the tensor decomposition.
---
paper_title: Function factorization using warped Gaussian processes

paper_content:

We introduce a new approach to non-linear regression called function factorization, that is suitable for problems where an output variable can reasonably be modeled by a number of multiplicative interaction terms between non-linear functions of the inputs. The idea is to approximate a complicated function on a high-dimensional space by the sum of products of simpler functions on lower-dimensional subspaces. Function factorization can be seen as a generalization of matrix and tensor factorization methods, in which the data are approximated by the sum of outer products of vectors. We present a non-parametric Bayesian approach to function factorization where the priors over the factorizing functions are warped Gaussian processes, and we do inference using Hamiltonian Markov chain Monte Carlo. We demonstrate the superior predictive performance of the method on a food science data set compared to Gaussian process regression and tensor factorization using PARAFAC and GEMANOVA models.
---
paper_title: Unsupervised single-channel source separation using bayesian NMF

paper_content:

We propose a prior structure for single-channel audio source separation using Non-Negative Matrix Factorisation. For the tonal and percussive signals, the model assigns different prior distributions to the corresponding parts of the template and excitation matrices. This partitioning enables not only more realistic modelling, but also a deterministic way to group the components into sources. This also prevents the possibility of not detecting/assigning a component and remove the need for a dataset and training. Our method only needs the number of components of each source to be set, but this does not play a crucial role in the performance. Very promising results can be obtained using the model with too few design decisions and moderate time complexity.
---
paper_title: Probabilistic modeling paradigms for audio source separation

paper_content:

Most sound scenes result from the superposition of several sources, which can be separately perceived and analyzed by human listeners. Source separation aims to provide machine listeners with similar skills by extracting the sounds of individual sources from a given scene. Existing separation systems operate either by emulating the human auditory system or by inferring the parameters of probabilistic sound models. In this chapter, we focus on the latter approach and provide a joint overview of established and recent models, including independent component analysis, local time-frequency models and spectral template-based models. We show that most models are instances of one of the following two general paradigms: linear modeling or variance modeling. We compare the merits of either paradigm and report objective performance figures. We conclude by discussing promising combinations of probabilistic priors and inference algorithms that could form the basis of future state-of-the-art systems.
---
paper_title: Sparse Representations in Audio and Music: From Coding to Source Separation

paper_content:

Sparse representations have proved a powerful tool in the analysis and processing of audio signals and already lie at the heart of popular coding standards such as MP3 and Dolby AAC. In this paper we give an overview of a number of current and emerging applications of sparse representations in areas from audio coding, audio enhancement and music transcription to blind source separation solutions that can solve the ?cocktail party problem.? In each case we will show how the prior assumption that the audio signals are approximately sparse in some time-frequency representation allows us to address the associated signal processing task.
---
paper_title: REpeating Pattern Extraction Technique (REPET): A Simple Method for Music/Voice Separation

paper_content:

Repetition is a core principle in music. Many musical pieces are characterized by an underlying repeating structure over which varying elements are superimposed. This is especially true for pop songs where a singer often overlays varying vocals on a repeating accompaniment. On this basis, we present the REpeating Pattern Extraction Technique (REPET), a novel and simple approach for separating the repeating “background” from the non-repeating “foreground” in a mixture. The basic idea is to identify the periodically repeating segments in the audio, compare them to a repeating segment model derived from them, and extract the repeating patterns via time-frequency masking. Experiments on data sets of 1,000 song clips and 14 full-track real-world songs showed that this method can be successfully applied for music/voice separation, competing with two recent state-of-the-art approaches. Further experiments showed that REPET can also be used as a preprocessor to pitch detection algorithms to improve melody extraction.
---
paper_title: Singing-voice separation from monaural recordings using robust principal component analysis

paper_content:

Separating singing voices from music accompaniment is an important task in many applications, such as music information retrieval, lyric recognition and alignment. Music accompaniment can be assumed to be in a low-rank subspace, because of its repetition structure; on the other hand, singing voices can be regarded as relatively sparse within songs. In this paper, based on this assumption, we propose using robust principal component analysis for singing-voice separation from music accompaniment. Moreover, we examine the separation result by using a binary time-frequency masking method. Evaluations on the MIR-1K dataset show that this method can achieve around 1∼1.4 dB higher GNSDR compared with two state-of-the-art approaches without using prior training or requiring particular features.
---
paper_title: Multichannel nonnegative tensor factorization with structured constraints for user-guided audio source separation

paper_content:

Separating multiple tracks from professionally produced music recordings (PPMRs) is still a challenging problem. We address this task with a user-guided approach in which the separation system is provided segmental information indicating the time activations of the particular instruments to separate. This information may typically be retrieved from manual annotation. We use a so-called multichannel nonnegative tensor factorization (NTF) model, in which the original sources are observed through a multichannel convolutive mixture and in which the source power spectrograms are jointly modeled by a 3-valence (time/frequency/source) tensor. Our user-guided separation method produced competitive results at the 2010 Signal Separation Evaluation Campaign, with sufficient quality for real-world music editing applications.
---
paper_title: Score informed audio source separation using a parametric model of non-negative spectrogram

paper_content:

In this paper we present a new technique for monaural source separation in musical mixtures, which uses the knowledge of the musical score. This information is used to initialize an algorithm which computes a parametric decomposition of the spectrogram based on non-negative matrix factorization (NMF). This algorithm provides time-frequency masks which are used to separate the sources with Wiener filtering.
---
paper_title: Musical Audio Source Separation Based on User-Selected F 0 Track ?

paper_content:

A system for user-guided audio source separation is presented in this article. Following previous works on time-frequency music representations, the proposed User Interface allows the user to select the desired audio source, by means of the assumed fundamental frequency (F0) track of that source. The system then automatically refines the selected F0 tracks, estimates and separates the corresponding source from the mixture. The interface was tested and the separation results compare positively to the results of a fully automatic system, showing that the F0 track selection improves the separation performance.
---
paper_title: Blind Harmonic Adaptive Decomposition applied to supervised source separation

paper_content:

In this paper, a new supervised source separation system is introduced. The Constant-Q Transform (CQT) of an audio signal is first analyzed through an algorithm called Blind Harmonic Adaptive Decomposition (BHAD). This algorithm provides an estimation of the polyphonic pitch content of the input signal, from which the user can select the notes to be extracted. The system then automatically separates the corresponding source from the audio mixture, by means of time-frequency masking of the CQT. The system has been evaluated both in a task of multipitch estimation in order to measure the quality of the decomposition, and in a task of user-guided melody extraction to assess the quality of the separation. The very promising results obtained highlight the reliability of the proposed model.
---
paper_title: Score guided musical source separation using Generalized Coupled Tensor Factorization

paper_content:

Providing prior knowledge about sources to guide source separation is known to be useful in many audio applications. In this paper we present two tensor factorization models for musical source separation where musical information is incorporated by using the Generalized Coupled Tensor Factorization (GCTF) framework. The approach is an extension of Nonnegative Matrix Factorization where more than one matrix or tensor object is simultaneously factorized. The first model uses a temporally aligned transcription of the mixture and incorporates spectral knowledge via coupling. In contrast of using a temporally aligned transcription, the second model incorporates harmonic information by taking an approximate, incomplete, and not necessarily aligned transcription of the musical piece as input. We evaluate our models on piano and cello duets where the experiments show that instead of using a temporally aligned transcription, we can achieve competitive results by using only a partial and incomplete transcription.
---
paper_title: Algorithms for probabilistic latent tensor factorization

paper_content:

We propose a general probabilistic framework for modelling multiway data. Our approach establishes a novel link between graphical representation of probability measures and tensor factorization models that allow us to design arbitrary tensor factorization models while retaining simplicity. Using an expectation-maximization (EM) approach for maximizing the likelihood of the exponential dispersion models (EDM), we obtain iterative update equations for Kullback-Leibler (KL), Euclidian (EU) or Itakura-Saito (IS) costs as special cases. Besides EM, we derive alternative algorithms with multiplicative update rules (MUR) and alternating projections. We also provide algorithms for MAP estimation with conjugate priors. All of the algorithms can be formulated as message passing algorithm on a graph where vertices correspond to indices and cliques represent factors of the tensor decomposition.
---
paper_title: Score guided musical source separation using Generalized Coupled Tensor Factorization

paper_content:

Providing prior knowledge about sources to guide source separation is known to be useful in many audio applications. In this paper we present two tensor factorization models for musical source separation where musical information is incorporated by using the Generalized Coupled Tensor Factorization (GCTF) framework. The approach is an extension of Nonnegative Matrix Factorization where more than one matrix or tensor object is simultaneously factorized. The first model uses a temporally aligned transcription of the mixture and incorporates spectral knowledge via coupling. In contrast of using a temporally aligned transcription, the second model incorporates harmonic information by taking an approximate, incomplete, and not necessarily aligned transcription of the musical piece as input. We evaluate our models on piano and cello duets where the experiments show that instead of using a temporally aligned transcription, we can achieve competitive results by using only a partial and incomplete transcription.
---
paper_title: Convolutive common audio signal extraction

paper_content:

This paper addresses the extraction of a common signal among several mono audio tracks when this common signal undergoes a track-specific filtering. This problem arises in the extraction of a common music and effects track from a set of soundtracks in different languages. To this aim, a novel approach is proposed. The method is based on the dictionary modeling of track-specific and common signals, and is compared to a previous one proposed by the authors based on geometric considerations. The approach is integrated into a Non-Negative Matrix Factorization framework using the Itakura-Saito divergence. The method is evaluated on a synthetic database composed of filtered music and effects tracks, the filters being track-specific, and track-specific dialogs. The results show that this task becomes tractable, while the previously introduced method could not handle track-specific filtering.
---
paper_title: Separation by “humming”: User-guided sound extraction from monophonic mixtures

paper_content:

In this paper we present a novel approach for isolating and removing sounds from dense monophonic mixtures. The approach is user-based, and requires the presentation of a guide sound that mimics the desired target the user wishes to extract. The guide sound can be simply produced from a user by vocalizing or otherwise replicating the target sound marked for separation. Using that guide as a prior in a statistical sound mixtures model, we propose a methodology that allows us to efficiently extract complex structured sounds from dense mixtures.
---
paper_title: Separation of Music+Effects Sound Track from Several International Versions of the Same Movie

paper_content:

This article concerns the separation of the music+effects (ME) track from a movie sountrack, given the observation of several international versions of the same movie. The approach chosen is strongly inspired from existing stereo audio source separation and especially from spatial filtering algorithms such as DUET that can extract a constant panned source from a mixture very efficiently. The problem is indeed similar for we aim here at separating the ME track, which is the common background of all international versions of the movie soundtrack. The algorithm has been adapted to a number of channels greater than 2. Preprocessing techniques have also been proposed to adapt the algorithm to realistic cases. The performances of the algorithm have been evaluated on realistic and synthetic cases.
---
paper_title: A Watermarking-Based Method for Informed Source Separation of Audio Signals With a Single Sensor

paper_content:

In this paper, the issue of audio source separation from a single channel is addressed, i.e., the estimation of several source signals from a single observation of their mixture. This challenging problem is tackled with a specific two levels coder-decoder configuration. At the coder, source signals are assumed to be available before the mix is processed. Each source signal is characterized by a set of parameters that provide additional information useful for separation. We propose an original method using a watermarking technique to imperceptibly embed this information about the source signals into the mix signal. At the decoder, the watermark is extracted from the mix signal to enable an end-user who has no access to the original sources to separate these signals from their mixture. Hence, we call this separation process informed source separation (ISS). Thereby, several instruments or voice signals can be segregated from a single piece of music to enable post-mixing processing such as volume control, echo addition, spatialization, or timbre transformation. Good performances are obtained for the separation of up to four source signals, from mixtures of speech or music signals. Promising results open up new perspectives in both under-determined source separation and audio watermarking domains.
---
paper_title: Informed source separation: Source coding meets source separation

paper_content:

We consider the informed source separation (ISS) problem where, given the sources and the mixtures, any kind of side-information can be computed during a so-called encoding stage. This side-information is then used to assist source separation, given the mixtures only, at the so-called decoding stage. State of the art ISS approaches do not really consider ISS as a coding problem and rely on some purely source separation-inspired strategies, leading to performances that can at best reach those of oracle estimators. On the other hand, classical source coding strategies are not optimal either, since they do not benefit from the mixture availability. We introduce a general probabilistic framework called coding-based ISS (CISS) that consists in quantizing the sources using some posterior source distribution from those usually used in probabilistic model-based source separation. CISS benefits from both source coding, thanks to the source quantization, and source separation, thanks to the use of the posterior distribution that depends on the mixture. Our experiments show that CISS based on a particular model considerably outperforms for all rates both the conventional ISS approach and the source coding approach based on the same model.
---
paper_title: Informed source separation through spectrogram coding and data embedding $

paper_content:

We address the issue of underdetermined source separation in a particular informed configuration where both the sources and the mixtures are known during a so-called encoding stage. This knowledge enables the computation of a side-information which is small enough to be inaudibly embedded into the mixtures. At the decoding stage, the sources are no longer assumed to be known, only the mixtures and the extracted side-information are processed for source separation. The proposed system models the sources as independent and locally stationary Gaussian processes (GP) and the mixing process as a linear filtering. This model allows reliable estimation of the sources through generalized Wiener filtering, provided their spectrograms are known. As these spectrograms are too large to be embedded in the mixtures, we show how they can be efficiently approximated using either Nonnegative Tensor Factorization (NTF) or image compression. A high-capacity embedding method is used by the system to inaudibly embed the separation side-information into the mixtures. This method is an application of the Quantization Index Modulation technique applied to the time-frequency coefficients of the mixtures and permits to reach embedding rates of about 250kbps. Finally, a study of the performance of the full system is presented.
---
paper_title: Spatial Audio Object Coding (SAOC) - The Upcoming MPEG Standard on Parametric Object Based Audio Coding

paper_content:

Following the recent trend of employing parametric enhancement tools for increasing coding or spatial rendering efficiency, Spatial Audio Object Coding (SAOC) is one of the recent standardization activities in the MPEG audio group. SAOC is a technique for efficient coding and flexible, user-controllable rendering of multiple audio objects based on transmission of a mono or stereo downmix of the object signals. The SAOC system extends the MPEG Surround standard by re-using its spatial rendering capabilities. This paper will describe the chosen reference model architecture, the association between the different operational modes and applications, and the current status of the standardization process.
---

Title: An Overview of Informed Audio Source Separation

Section 1: INTRODUCTION
Description 1: Provide an introduction to audio source separation, its applications, challenges, and the concept of informed source separation.

Section 2: Notations
Description 2: Describe the notations and definitions used in the paper, particularly in the context of musical processing and audio signals.

Section 3: Gaussian model
Description 3: Discuss the Gaussian model for audio signals, including assumptions of local stationarity and the mathematical formulation of source separation.

Section 4: Separation procedure
Description 4: Explain the separation procedure, specifically the minimum mean-squared estimate (MMSE) and Wiener filtering, along with challenges in parameter estimation.

Section 5: MODEL-BASED SIDE-INFORMATION
Description 5: Describe the model-based approach to incorporating side-information for source separation, including specific models, prior distributions, and the handling of sparse and dense sources.

Section 6: SIGNAL-BASED SIDE-INFORMATION
Description 6: Discuss the signal-based approach, detailing the use of extra data such as score-sheets, MIDI files, and other related audio signals to guide the separation process.

Section 7: Score-informed source separation
Description 7: Elaborate on the techniques and challenges associated with using score sheets or MIDI files to enhance source separation.

Section 8: Exemplar-based source separation
Description 8: Describe the use of additional audio recordings, such as different versions of a song or user-provided covers, for improving source separation.

Section 9: Oracle source separation and spatial object coding
Description 9: Discuss advanced techniques like oracle source separation and Spatial Audio Object Coding (SAOC), including their potential in audio coding frameworks.

Section 10: CONCLUSION
Description 10: Summarize the key points discussed in the paper, emphasizing the benefits of informed source separation and highlighting future research directions.