Variational Variance: Simple, Reliable, Calibrated Heteroscedastic Noise Variance Parameterization
This addresses calibration issues in probabilistic models for researchers and practitioners, though it is incremental as it builds on prior work on regularization and PPCs.
The paper tackled the problem of brittle optimization in models like regression and VAEs when fitting both mean and variance, which harms likelihoods and fails posterior predictive checks. The authors proposed a simple variational treatment of heteroscedastic variance, which improved parameter calibration and sample quality while preserving or outperforming existing likelihoods.
Brittle optimization has been observed to adversely impact model likelihoods for regression and VAEs when simultaneously fitting neural network mappings from a (random) variable onto the mean and variance of a dependent Gaussian variable. Previous works have bolstered optimization and improved likelihoods, but fail other basic posterior predictive checks (PPCs). Under the PPC framework, we propose critiques to test predictive mean and variance calibration and the predictive distribution's ability to generate sensible data. We find that our attractively simple solution, to treat heteroscedastic variance variationally, sufficiently regularizes variance to pass these PPCs. We consider a diverse gamut of existing and novel priors and find our methods preserve or outperform existing model likelihoods while significantly improving parameter calibration and sample quality for regression and VAEs.