A Heavy-Tailed Algebra for Probabilistic Programming
This work addresses a specific bottleneck in probabilistic programming for researchers and practitioners by improving tail behavior modeling, though it is incremental as it builds on existing methods with a novel algebraic framework.
The paper tackles the problem of probabilistic models failing to accurately capture tail behavior by proposing a systematic approach for analyzing tails of random variables using a three-parameter algebra based on the generalized Gamma distribution. The result shows that inference algorithms leveraging this heavy-tailed algebra achieve superior performance in density modeling and variational inference tasks.
Despite the successes of probabilistic models based on passing noise through neural networks, recent work has identified that such methods often fail to capture tail behavior accurately, unless the tails of the base distribution are appropriately calibrated. To overcome this deficiency, we propose a systematic approach for analyzing the tails of random variables, and we illustrate how this approach can be used during the static analysis (before drawing samples) pass of a probabilistic programming language compiler. To characterize how the tails change under various operations, we develop an algebra which acts on a three-parameter family of tail asymptotics and which is based on the generalized Gamma distribution. Our algebraic operations are closed under addition and multiplication; they are capable of distinguishing sub-Gaussians with differing scales; and they handle ratios sufficiently well to reproduce the tails of most important statistical distributions directly from their definitions. Our empirical results confirm that inference algorithms that leverage our heavy-tailed algebra attain superior performance across a number of density modeling and variational inference tasks.