Evidential Softmax for Sparse Multimodal Distributions in Deep Generative Models
This addresses a computational bottleneck for researchers and practitioners using generative models in applications requiring marginalization, though it is an incremental improvement over existing sparse normalization methods.
The paper tackled the problem of sparse normalization functions in deep generative models collapsing multimodality, and introduced ev-softmax, which preserves multimodality while enabling exact marginalization, outperforming existing techniques in distributional accuracy.
Many applications of generative models rely on the marginalization of their high-dimensional output probability distributions. Normalization functions that yield sparse probability distributions can make exact marginalization more computationally tractable. However, sparse normalization functions usually require alternative loss functions for training since the log-likelihood is undefined for sparse probability distributions. Furthermore, many sparse normalization functions often collapse the multimodality of distributions. In this work, we present $\textit{ev-softmax}$, a sparse normalization function that preserves the multimodality of probability distributions. We derive its properties, including its gradient in closed-form, and introduce a continuous family of approximations to $\textit{ev-softmax}$ that have full support and can be trained with probabilistic loss functions such as negative log-likelihood and Kullback-Leibler divergence. We evaluate our method on a variety of generative models, including variational autoencoders and auto-regressive architectures. Our method outperforms existing dense and sparse normalization techniques in distributional accuracy. We demonstrate that $\textit{ev-softmax}$ successfully reduces the dimensionality of probability distributions while maintaining multimodality.