Step-size Optimization for Continual Learning
This addresses the challenge of optimizing step-sizes for neural networks in continual learning, though it is incremental as it builds on existing meta-gradient methods.
The paper tackled the problem of step-size adaptation in continual learning by comparing heuristic methods like RMSProp and Adam with stochastic meta-gradient descent (IDBD), showing that IDBD consistently improves step-size vectors on simple problems where the heuristics fail.
In continual learning, a learner has to keep learning from the data over its whole life time. A key issue is to decide what knowledge to keep and what knowledge to let go. In a neural network, this can be implemented by using a step-size vector to scale how much gradient samples change network weights. Common algorithms, like RMSProp and Adam, use heuristics, specifically normalization, to adapt this step-size vector. In this paper, we show that those heuristics ignore the effect of their adaptation on the overall objective function, for example by moving the step-size vector away from better step-size vectors. On the other hand, stochastic meta-gradient descent algorithms, like IDBD (Sutton, 1992), explicitly optimize the step-size vector with respect to the overall objective function. On simple problems, we show that IDBD is able to consistently improve step-size vectors, where RMSProp and Adam do not. We explain the differences between the two approaches and their respective limitations. We conclude by suggesting that combining both approaches could be a promising future direction to improve the performance of neural networks in continual learning.