Towards Self-Supervised Covariance Estimation in Deep Heteroscedastic Regression
This work addresses the challenge of covariance estimation in heteroscedastic regression, which is important for applications requiring uncertainty quantification, but it is incremental as it builds on existing unsupervised frameworks with a new self-supervised twist.
The paper tackles the problem of estimating covariance in deep heteroscedastic regression by proposing a self-supervised method that uses a derived upper bound on the 2-Wasserstein distance and a neighborhood-based heuristic for pseudo labels, resulting in a computationally cheaper and accurate approach as demonstrated on synthetic and real datasets.
Deep heteroscedastic regression models the mean and covariance of the target distribution through neural networks. The challenge arises from heteroscedasticity, which implies that the covariance is sample dependent and is often unknown. Consequently, recent methods learn the covariance through unsupervised frameworks, which unfortunately yield a trade-off between computational complexity and accuracy. While this trade-off could be alleviated through supervision, obtaining labels for the covariance is non-trivial. Here, we study self-supervised covariance estimation in deep heteroscedastic regression. We address two questions: (1) How should we supervise the covariance assuming ground truth is available? (2) How can we obtain pseudo labels in the absence of the ground-truth? We address (1) by analysing two popular measures: the KL Divergence and the 2-Wasserstein distance. Subsequently, we derive an upper bound on the 2-Wasserstein distance between normal distributions with non-commutative covariances that is stable to optimize. We address (2) through a simple neighborhood based heuristic algorithm which results in surprisingly effective pseudo labels for the covariance. Our experiments over a wide range of synthetic and real datasets demonstrate that the proposed 2-Wasserstein bound coupled with pseudo label annotations results in a computationally cheaper yet accurate deep heteroscedastic regression.