A Bridge Between Hyperparameter Optimization and Learning-to-learn
This work addresses a foundational problem in machine learning by bridging hyperparameter optimization and meta-learning, but it appears incremental as it builds on existing gradient-based methods.
The paper tackles the problem of unifying gradient-based hyperparameter optimization and meta-learning by considering nested optimization problems with inner and outer objectives, showing that existing methods can be instantiated within this framework and presenting preliminary experiments for few-shot learning.
We consider a class of a nested optimization problems involving inner and outer objectives. We observe that by taking into explicit account the optimization dynamics for the inner objective it is possible to derive a general framework that unifies gradient-based hyperparameter optimization and meta-learning (or learning-to-learn). Depending on the specific setting, the variables of the outer objective take either the meaning of hyperparameters in a supervised learning problem or parameters of a meta-learner. We show that some recently proposed methods in the latter setting can be instantiated in our framework and tackled with the same gradient-based algorithms. Finally, we discuss possible design patterns for learning-to-learn and present encouraging preliminary experiments for few-shot learning.