Squared Neural Families: A New Class of Tractable Density Models
This provides a flexible and tractable density model for machine learning tasks involving probability distributions, representing a novel paradigm rather than an incremental improvement.
The authors introduced Squared Neural Families (SNEFY), a new class of tractable probability distributions formed by squaring neural network norms, which generalizes exponential families and allows closed-form normalizing constants. They demonstrated its utility on tasks like density estimation and conditional density estimation, showing flexibility and tractability.
Flexible models for probability distributions are an essential ingredient in many machine learning tasks. We develop and investigate a new class of probability distributions, which we call a Squared Neural Family (SNEFY), formed by squaring the 2-norm of a neural network and normalising it with respect to a base measure. Following the reasoning similar to the well established connections between infinitely wide neural networks and Gaussian processes, we show that SNEFYs admit closed form normalising constants in many cases of interest, thereby resulting in flexible yet fully tractable density models. SNEFYs strictly generalise classical exponential families, are closed under conditioning, and have tractable marginal distributions. Their utility is illustrated on a variety of density estimation, conditional density estimation, and density estimation with missing data tasks.