Learnable Bernoulli Dropout for Bayesian Deep Learning
This work addresses the need for better uncertainty estimation and performance in Bayesian deep learning, particularly for applications like image analysis and recommendation systems, though it is incremental as it builds on existing dropout and VAE methods.
The authors tackled the problem of improving prediction robustness and uncertainty quantification in deep learning by proposing learnable Bernoulli dropout (LBD), a model-agnostic dropout scheme with jointly optimized dropout rates, which led to superior accuracy and uncertainty estimates in tasks like image classification and semantic segmentation, and achieved state-of-the-art performance on collaborative filtering datasets.
In this work, we propose learnable Bernoulli dropout (LBD), a new model-agnostic dropout scheme that considers the dropout rates as parameters jointly optimized with other model parameters. By probabilistic modeling of Bernoulli dropout, our method enables more robust prediction and uncertainty quantification in deep models. Especially, when combined with variational auto-encoders (VAEs), LBD enables flexible semi-implicit posterior representations, leading to new semi-implicit VAE~(SIVAE) models. We solve the optimization for training with respect to the dropout parameters using Augment-REINFORCE-Merge (ARM), an unbiased and low-variance gradient estimator. Our experiments on a range of tasks show the superior performance of our approach compared with other commonly used dropout schemes. Overall, LBD leads to improved accuracy and uncertainty estimates in image classification and semantic segmentation. Moreover, using SIVAE, we can achieve state-of-the-art performance on collaborative filtering for implicit feedback on several public datasets.