Particle Filtering Methods for Stochastic Optimization with Application to Large-Scale Empirical Risk Minimization
This work addresses large-scale empirical risk minimization for machine learning practitioners, offering an incremental improvement by adapting particle filtering to stochastic optimization.
The paper tackled the problem of stochastic optimization by developing particle filtering-based stochastic optimizers (PFSOs) to overcome limitations of existing methods like Kalman filter-based approaches, resulting in improved numerical stability, convergence speed, and flexibility in handling diverse loss functions, as demonstrated through experiments on simulated and real datasets.
This paper is concerned with sequential filtering based stochastic optimization (FSO) approaches that leverage a probabilistic perspective to implement the incremental proximity method (IPM). The present FSO methods are derived based on the Kalman filter (KF) and the extended KF (EKF). In contrast with typical methods such as stochastic gradient descent (SGD) and IPMs, they do not need to pre-schedule the learning rate for convergence. Nevertheless, they have limitations that inherit from the KF mechanism. As the particle filtering (PF) method outperforms KF and its variants remarkably for nonlinear non-Gaussian sequential filtering problems, it is natural to ask if FSO methods can benefit from PF to get around of their limitations. We provide an affirmative answer to this question by developing two PF based stochastic optimizers (PFSOs). For performance evaluation, we apply them to address nonlinear least-square fitting with simulated data, and empirical risk minimization for binary classification of real data sets. Experimental results demonstrate that PFSOs outperform remarkably a benchmark SGD algorithm, the vanilla IPM, and KF-type FSO methods in terms of numerical stability, convergence speed, and flexibility in handling diverse types of loss functions.