Ensemble Kalman Inversion: A Derivative-Free Technique For Machine Learning Tasks
This work addresses the need for gradient-free optimization techniques in machine learning, particularly for tasks where gradient computation is challenging, though it appears incremental as it adapts existing EKI methods to ML.
The authors tackled the problem of solving machine learning tasks without requiring gradient computations by reformulating them as inverse problems and proposing a derivative-free algorithm based on ensemble Kalman inversion (EKI). The method demonstrated wide applicability and robustness in numerical experiments, including supervised and semi-supervised learning with deep neural networks.
The standard probabilistic perspective on machine learning gives rise to empirical risk-minimization tasks that are frequently solved by stochastic gradient descent (SGD) and variants thereof. We present a formulation of these tasks as classical inverse or filtering problems and, furthermore, we propose an efficient, gradient-free algorithm for finding a solution to these problems using ensemble Kalman inversion (EKI). Applications of our approach include offline and online supervised learning with deep neural networks, as well as graph-based semi-supervised learning. The essence of the EKI procedure is an ensemble based approximate gradient descent in which derivatives are replaced by differences from within the ensemble. We suggest several modifications to the basic method, derived from empirically successful heuristics developed in the context of SGD. Numerical results demonstrate wide applicability and robustness of the proposed algorithm.