A Heavy-Tailed Algebra for Probabilistic Programming
Feynman Liang Liam Hodgkinson Michael W. Mahoney
Despite the successes of probabilistic models based on passing noise through
neural networks, recent work has identified that such methods often fail to
capture tail behavior accurately, unless the tails of the base distribution are
appropriately calibrated. To overcome this deficiency, we propose a systematic
approach for analyzing the tails of random variables, and we illustrate how
this approach can be used during the static analysis (before drawing samples)
pass of a probabilistic programming language compiler. To characterize how the
tails change under various operations, we develop an algebra which acts on a
three-parameter family of tail asymptotics and which is based on the
generalized Gamma distribution. Our algebraic operations are closed under
addition and multiplication; they are capable of distinguishing sub-Gaussians
with differing scales; and they handle ratios sufficiently well to reproduce
the tails of most important statistical distributions directly from their
definitions. Our empirical results confirm that inference algorithms that
leverage our heavy-tailed algebra attain superior performance across a number
of density modeling and variational inference tasks.