Online Multi-Task Learning with Recursive Least Squares and Recursive Kernel Methods
This work addresses online multi-task learning for regression problems, offering more efficient alternatives to existing methods like online gradient descent, but it is incremental as it builds on known graph-based formulations.
The paper tackles online multi-task learning regression by introducing two recursive methods, MT-WRLS and MT-OSLSSVR, which achieve exact or approximate recursions with quadratic per-instance cost, and demonstrates significant performance gains in a wind speed forecasting case study.
This paper introduces two novel approaches for Online Multi-Task Learning (MTL) Regression Problems. We employ a high performance graph-based MTL formulation and develop two alternative recursive versions based on the Weighted Recursive Least Squares (WRLS) and the Online Sparse Least Squares Support Vector Regression (OSLSSVR) strategies. Adopting task-stacking transformations, we demonstrate the existence of a single matrix incorporating the relationship of multiple tasks and providing structural information to be embodied by the MT-WRLS method in its initialization procedure and by the MT-OSLSSVR in its multi-task kernel function. Contrasting the existing literature, which is mostly based on Online Gradient Descent (OGD) or cubic inexact approaches, we achieve exact and approximate recursions with quadratic per-instance cost on the dimension of the input space (MT-WRLS) or on the size of the dictionary of instances (MT-OSLSSVR). We compare our online MTL methods to other contenders in a real-world wind speed forecasting case study, evidencing the significant gain in performance of both proposed approaches.