Within many machine learning algorithms, a fundamental problem concerns efficient calculation of an unbiased gradient wrt parameters $\boldsymbol{\gamma}$ for expectation-based objectives $\mathbb{E}_{q_{\boldsymbol{\gamma}} (\boldsymbol{y})} [f (\boldsymbol{y}) ]$. Most existing methods either ($i$) suffer from high variance, seeking help from (often) complicated variance-reduction techniques; or ($ii$) they only apply to reparameterizable continuous random variables and employ a reparameterization trick. To address these limitations, we propose a General and One-sample (GO) gradient that ($i$) applies to many distributions associated with non-reparameterizable continuous {\em or} discrete random variables, and ($ii$) has the same low-variance as the reparameterization trick. We find that the GO gradient often works well in practice based on only one Monte Carlo sample (although one can of course use more samples if desired). Alongside the GO gradient, we develop a means of propagating the chain rule through distributions, yielding statistical back-propagation, coupling neural networks to common random variables. Neural networks, typically trained using back-propagation for parameter optimization, have recently demonstrated significant success across a wide range of applications. There has been interest in coupling neural networks with random variables, so as to embrace greater descriptive capacity. Recent examples of this include black-box variational inference (BBVI) BID17 BID33 BID29 BID11 BID32 BID31 Zhang et al., 2018) and generative adversarial networks (GANs) BID8 BID28 Zhao et al., 2016; BID1 BID20 . Unfortunately, efficiently backpropagating gradients through general distributions (random variables) remains a bottleneck. Most current methodology focuses on distributions with continuous random variables, for which the reparameterization trick may be readily applied BID17 BID9 .As an example, the aforementioned bottleneck greatly constrains the applicability of BBVI, by limiting variational approximations to reparameterizable distributions. This limitation excludes discrete random variables and many types of continuous ones. From the perspective of GAN, the need to employ reparameterization has constrained most applications to continuous observations. There are many forms of data that are more-naturally discrete.The fundamental problem associated with the aforementioned challenges is the need to efficiently calculate an unbiased low-variance gradient wrt parameters γ for an expectation objective of the form E qγ (y) [f (y)]1 . We are interested in general distributions q γ (y), for which the components of y may be either continuous or discrete. Typically the components of y have a hierarchical structure, and a subset of the components of y play a role in evaluating f (y).Unfortunately , classical methods for estimating gradients of E qγ (y) [f (y)] wrt γ have limitations. The REINFORCE gradient (Williams, 1992) , although generally applicable (e.g., for continuous and discrete random variables), exhibits high variance with Monte Carlo (MC) estimation of the expectation, forcing one to apply additional variance-reduction techniques. The reparameterization trick (Rep) BID38 BID17 BID33 works well, with as few as only one MC sample, but it is limited to continuous reparameterizable y. Many efforts have been devoted to improving these two formulations, as detailed in Section 6. However, none of these methods is characterized by generalization (applicable to general distributions) and efficiency (working well with as few as one MC sample).The key contributions of this work are based on the recognition that REINFORCE and Rep are seeking to solve the same objective, but in practice Rep yields lower-variance estimations, albeit for a narrower class of distributions. Recent work BID32 has made a connection between REINFORCE and Rep, recognizing that the former estimates a term the latter evaluates analytically. The high variance by which REINFORCE approximates this term manifests high variance in the gradient estimation. Extending these ideas, we make the following main contributions. (i) We propose a new General and One-sample (GO) gradient in Section 3, that principally generalizes Rep to many non-reparameterizable distributions and justifies two recent methods BID5 BID15 ; the "One sample" motivating the name GO is meant to highlight the low variance of the proposed method, although of course one may use more than one sample if desired. (ii) We find that the core of the GO gradient is something we term a variable-nabla, which can be interpreted as the gradient of a random variable wrt a parameter. (iii) Utilizing variablenablas to propagate the chain rule through distributions, we broaden the applicability of the GO gradient in Sections 4-5 and present statistical back-propagation, a statistical generalization of classic back-propagation BID36 . Through this generalization, we may couple neural networks to general random variables, and compute needed gradients with low variance. For expectation-based objectives, we propose a General and One-sample (GO) gradient that applies to continuous and discrete random variables. We further generalize the GO gradient to cases for which the underlying model is deep and has a marginal distribution corresponding to the latent variables of interest, and to cases for which the latent variables are hierarchical. The GO-gradient setup is demonstrated to yield the same low-variance estimation as the reparameterization trick, which is only applicable to reparameterizable continuous random variables. Alongside the GO gradient, we constitute a means of propagating the chain rule through distributions. Accordingly, we present statistical back-propagation, to flexibly integrate deep neural networks with general classes of random variables. A PROOF OF THEOREM 1We first prove (7) in the main manuscript, followed by its discrete counterpart, i.e., (8) in the main manuscript. Then, it is easy to verify Theorem 1.A.1 PROOF OF EQUATION FORMULA3 IN THE MAIN MANUSCRIPT Similar proof in one-dimension is also given in the supplemental materials of BID32 .We want to calculate DISPLAYFORM0 where y −v denotes y with y v excluded. Without loss of generality, we assume y v ∈ (−∞, ∞). DISPLAYFORM1 , and we have DISPLAYFORM2 where DISPLAYFORM3 , we then apply integration by parts (or partial integration) to get DISPLAYFORM4 With Q γ (∞) = 1 and Q γ (−∞) = 0, it's straightforward to verify that the first term is always zero for any Q γ (y v ), thus named the "0" term.
a Rep-like gradient for non-reparameterizable continuous/discrete distributions; further generalized to deep probabilistic models, yielding statistical back-propagation Presents a gradient estimator for expectation-based objectives that is unbiased, has low variance, and applies to either continuous and discrete random variables. An improved method for computing derivates of the expectation, and a new gradient estimator of low variance that allows training of generative models in which observations or latent variables are discrete. Designs a low variance gradient for distributions associated with continuous or discrete random variables.