Robust Frequent Directions with Application in Online Learning
This work addresses a specific bottleneck in online sketching for machine learning, offering an incremental improvement with practical benefits for online learning applications.
The authors tackled the rank deficiency problem in the frequent directions sketching technique by introducing robust frequent directions (RFD) with a regularization term, which reduces approximation error without extra computational cost and leads to a hyperparameter-free online Newton algorithm that outperforms state-of-the-art second-order online learning methods in experiments.
The frequent directions (FD) technique is a deterministic approach for online sketching that has many applications in machine learning. The conventional FD is a heuristic procedure that often outputs rank deficient matrices. To overcome the rank deficiency problem, we propose a new sketching strategy called robust frequent directions (RFD) by introducing a regularization term. RFD can be derived from an optimization problem. It updates the sketch matrix and the regularization term adaptively and jointly. RFD reduces the approximation error of FD without increasing the computational cost. We also apply RFD to online learning and propose an effective hyperparameter-free online Newton algorithm. We derive a regret bound for our online Newton algorithm based on RFD, which guarantees the robustness of the algorithm. The experimental studies demonstrate that the proposed method outperforms state-of-the-art second order online learning algorithms.