The maximum mean discrepancy (MMD) between two probability measures P
 and Q is a metric that is zero if and only if all moments of the two measures
 are equal, making it an appealing statistic for two-sample tests. Given i.i.d. samples
 from P and Q, Gretton et al. (2012) show that we can construct an unbiased
 estimator for the square of the MMD between the two distributions. If P is a
 distribution of interest and Q is the distribution implied by a generative neural
 network with stochastic inputs, we can use this estimator to train our neural network.
 However, in practice we do not always have i.i.d. samples from our target
 of interest. Data sets often exhibit biases—for example, under-representation of
 certain demographics—and if we ignore this fact our machine learning algorithms
 will propagate these biases. Alternatively, it may be useful to assume our data has
 been gathered via a biased sample selection mechanism in order to manipulate
 properties of the estimating distribution Q.
 In this paper, we construct an estimator for the MMD between P and Q when we
 only have access to P via some biased sample selection mechanism, and suggest
 methods for estimating this sample selection mechanism when it is not already
 known. We show that this estimator can be used to train generative neural networks
 on a biased data sample, to give a simulator that reverses the effect of that
 bias. Neural networks with stochastic input layers can be trained to approximately sample from an arbitrary probability distribution P based on samples from P BID7 . Generating simulations from complex distributions has applications in a large number of fields: We can automatically generate illustrations for text BID21 or streams of video BID20 ; we can simulate novel molecular fingerprints to aid scientific exploration BID11 ; and, we can synthesize medical time-series data that can be shared without violating patient privacy BID6 .In this paper, we consider the setting of a feedforward neural network (referred to as the generator) that maps random noise inputs z ∈ R d to some observation space X . The weights of the neural network are trained to minimize some loss function between the resulting simulations and exemplars of real data. The general form of the resulting distribution Q over simulations G(z) is determined by the architecture of the generator-which governs the form of the mapping G-and by the loss function used to train the generator. Generative adversarial networks BID7 use dynamically varying, adversarially learned loss functions specified in terms of the output of a classifier. Other generative networks use a loss function defined using a distributional distance or divergence between the simulation distribution Q and a target distribution P BID0 BID15 BID22 , requiring the generator to mimic the variance in a collection of data points rather than simply converge to a single mode. In particular, the maximum mean discrepancy BID8 has demonstrated good performance as a loss function in this setting BID18 BID19 , since it reduces to zero if and only if all moments of two distributions are equal, requiring the generator to reproduce the full range of variation found in the data.These approaches, like most machine learning methods, assume our data is a representative sample from the distribution of interest. If this assumption is correct, minimizing the distributional distance between the simulations and the data is equivalent to learning a distribution that is indistinguishable under an appropriate two-sample test from our target distribution. However, we run into problems if our data is not in fact a representative sample from our target distribution-for instance, if our data gathering mechanism is susceptible to sample selection bias. The problem of machine learning algorithms replicating and even magnifying human biases is gathering increasing awareness BID2 BID23 , and if we believe our dataset suffers from such biases-for example, if our audio dataset contains primarily male speakers or our image dataset contains primarily white faces -we will typically want to take steps to correct this bias.Even if our data is representative of the underlying distribution, we might want to generate samples from a modified version of this distribution. For example, we might want to alter the demographics of characters in a scene to fit a story-line. In this setting, we can treat our desired modified distribution as our target distribution, and treat our data as if they were sampled from this distribution subject to an appropriately biased sample selection mechanism.If we know the form of our sample selection bias, we can reformulate our loss function to penalize the generator based on the difference between simulated data and the unbiased distribution of interest, which we will refer to as our target distribution. After a review of relevant background information in Section 2, we show in Section 3 that, given a function that describes how our observed data deviates from this target distribution, we can construct an estimator of the MMD between the generator and the target distribution.In practice, we will not know the function linking the target distribution and the empirical data distribution. However, we can approximate this function based on user-provided examples of data points that are over-and under-represented. In Section 4, we discuss ways to estimate this function, and in Section 5 we discuss related work in survey sampling statistics and bias reduction. We demonstrate the efficacy and applicability of our approach in Section 6. We have presented an asymptotically unbiased estimator for the MMD between two distributions P and Q, for use when we only have access to P via a biased sampling mechanism. This mechanism can be specified by a known or estimated thinning function T (x), where samples are then assumed to come from a distribution T (x)P(x)/Z. We show that this estimator can be used to manipulate the distribution of simulations learned by a generative network, in order to correct for sampling bias or to explicitly change the distribution according to a user-specified function.When the thinning function is unknown, it can be estimated from labeled data. We demonstrate this in an interpretable experiment using partially labeled images, where we jointly estimate the thinning function alongside the generator weights. An obvious next step is to explore the use of more sophisticated thinning functions appropriate for complex, multimodal settings.
We propose an estimator for the maximum mean discrepancy, appropriate when a target distribution is only accessible via a biased sample selection procedure, and show that it can be used in a generative network to correct for this bias. Proposes an importance-weighted estimator of the MMD to estimate the MMD between distributions based on samples biased according to a known or estimated unknown scheme. The authors address the problem of sample selection bias in MMD-GANs and propose an estimate of the MMD between two distributions using weighted maximum mean discrepancy. This paper presents a modification of the objective used to train generative networks with an MMD adversary 