Accelerated Learning with Robustness to Adversarial Regressors
This work addresses a critical need for robust learning algorithms in dynamic environments like control systems, offering both theoretical guarantees and practical applications, though it builds incrementally on existing momentum-based and variational methods.
The paper tackles the problem of designing parameter update algorithms that are stable and converge in the presence of adversarial time-varying regressors, common in control theory, by proposing a new discrete-time algorithm that provides stability guarantees and accelerated learning with convergence in at most $ ilde{\mathcal{O}}(1/\sqrt{\epsilon})$ iterations for constant regressors.
High order momentum-based parameter update algorithms have seen widespread applications in training machine learning models. Recently, connections with variational approaches have led to the derivation of new learning algorithms with accelerated learning guarantees. Such methods however, have only considered the case of static regressors. There is a significant need for parameter update algorithms which can be proven stable in the presence of adversarial time-varying regressors, as is commonplace in control theory. In this paper, we propose a new discrete time algorithm which 1) provides stability and asymptotic convergence guarantees in the presence of adversarial regressors by leveraging insights from adaptive control theory and 2) provides non-asymptotic accelerated learning guarantees leveraging insights from convex optimization. In particular, our algorithm reaches an $ε$ sub-optimal point in at most $\tilde{\mathcal{O}}(1/\sqrtε)$ iterations when regressors are constant - matching lower bounds due to Nesterov of $Ω(1/\sqrtε)$, up to a $\log(1/ε)$ factor and provides guaranteed bounds for stability when regressors are time-varying. We provide numerical experiments for a variant of Nesterov's provably hard convex optimization problem with time-varying regressors, as well as the problem of recovering an image with a time-varying blur and noise using streaming data.