Gradient-based optimization is the foundation of deep learning and reinforcement learning.
 Even when the mechanism being optimized is unknown or not differentiable, optimization using high-variance or biased gradient estimates is still often the best strategy. We introduce a general framework for learning low-variance, unbiased gradient estimators for black-box functions of random variables, based on gradients of a learned function.
 These estimators can be jointly trained with model parameters or policies, and are applicable in both discrete and continuous settings. We give unbiased, adaptive analogs of state-of-the-art reinforcement learning methods such as advantage actor-critic. We also demonstrate this framework for training discrete latent-variable models. Gradient-based optimization has been key to most recent advances in machine learning and reinforcement learning. The back-propagation algorithm BID21 , also known as reverse-mode automatic differentiation BID25 BID16 computes exact gradients of deterministic, differentiable objective functions. The reparameterization trick BID33 BID7 BID17 allows backpropagation to give unbiased, lowvariance estimates of gradients of expectations of continuous random variables. This has allowed effective stochastic optimization of large probabilistic latent-variable models.Unfortunately, there are many objective functions relevant to the machine learning community for which backpropagation cannot be applied. In reinforcement learning, for example, the function being optimized is unknown to the agent and is treated as a black box BID23 . Similarly, when fitting probabilistic models with discrete latent variables, discrete sampling operations create discontinuities giving the objective function zero gradient with respect to its parameters. Much recent work has been devoted to constructing gradient estimators for these situations. In reinforcement learning, advantage actor-critic methods BID27 give unbiased gradient estimates with reduced variance obtained by jointly optimizing the policy parameters with an estimate of the value function. In discrete latent-variable models, low-variance but biased gradient estimates can be given by continuous relaxations of discrete variables BID10 BID4 .A recent advance by BID30 used a continuous relaxation of discrete random variables to build an unbiased and lower-variance gradient estimator, and showed how to tune the free parameters of these relaxations to minimize the estimator's variance during training. We generalize the method of BID30 to learn a free-form control variate parameterized by a neural network. This gives a lower-variance, unbiased gradient estimator which can be applied to a wider variety of problems. Most notably, our method is applicable even when no continuous relaxation is available, as in reinforcement learning or black-box function optimization. In this work we synthesized and generalized several standard approaches for constructing gradient estimators. We proposed a generic gradient estimator that can be applied to expectations of known or black-box functions of discrete or continuous random variables, and adds little computational overhead. We also derived a simple extension to reinforcement learning in both discrete and continuous-action domains.Future applications of this method could include training models with hard attention or memory indexing BID36 . One could also apply our estimators to continuous latentvariable models whose likelihood is non-differentiable, such as a 3D rendering engine. Extensions to the reparameterization gradient estimator BID20 BID14 could also be applied to increase the scope of distributions that can be modeled.In the reinforcement learning setting, our method could be combined with other variance-reduction techniques such as generalized advantage estimation BID5 BID24 , or other optimization methods, such as KFAC BID35 . One could also train our control variate off-policy, as in Q-prop .
We present a general method for unbiased estimation of gradients of black-box functions of random variables. We apply this method to discrete variational inference and reinforcement learning.  Suggests a new approach to performing gradient descent for blackbox optimization or training discrete latent variable models.