Sample Average Approximation for Black-Box VI
This work addresses hyperparameter selection challenges in variational inference for machine learning practitioners, but it is incremental as it builds on existing approximation techniques.
The paper tackled the difficulties of stochastic gradient ascent in black-box variational inference by using sample average approximation to transform it into deterministic problems, achieving faster performance than existing methods.
We present a novel approach for black-box VI that bypasses the difficulties of stochastic gradient ascent, including the task of selecting step-sizes. Our approach involves using a sequence of sample average approximation (SAA) problems. SAA approximates the solution of stochastic optimization problems by transforming them into deterministic ones. We use quasi-Newton methods and line search to solve each deterministic optimization problem and present a heuristic policy to automate hyperparameter selection. Our experiments show that our method simplifies the VI problem and achieves faster performance than existing methods.