MaxEntropy Pursuit Variational Inference
This addresses a core problem in variational inference for machine learning practitioners, though it appears incremental as a variant of existing greedy approximation methods.
The paper tackles the trade-off between efficiency and accuracy in variational inference by proposing a greedy approximation method using Max-Entropy optimization, demonstrating its ability to capture complex multimodal posteriors in neural network continual learning.
One of the core problems in variational inference is a choice of approximate posterior distribution. It is crucial to trade-off between efficient inference with simple families as mean-field models and accuracy of inference. We propose a variant of a greedy approximation of the posterior distribution with tractable base learners. Using Max-Entropy approach, we obtain a well-defined optimization problem. We demonstrate the ability of the method to capture complex multimodal posterior via continual learning setting for neural networks.