Time-Smoothed Gradients for Online Forecasting
This work addresses the challenge of parameter tuning in online learning for forecasting applications, though it appears incremental.
The paper tackled the problem of learning rate sensitivity in stochastic gradient descent for online forecasting by proposing time-smoothed gradients, resulting in more stable and computationally efficient performance validated on the GEFCom2014 dataset.
Here, we study different update rules in stochastic gradient descent (SGD) for online forecasting problems. The selection of the learning rate parameter is critical in SGD. However, it may not be feasible to tune this parameter in online learning. Therefore, it is necessary to have an update rule that is not sensitive to the selection of the learning parameter. Inspired by the local regret metric that we introduced previously, we propose to use time-smoothed gradients within SGD update. Using the public data set-- GEFCom2014, we validate that our approach yields more stable results than the other existing approaches. Furthermore, we show that such a simple approach is computationally efficient compared to the alternatives.