Binomial Gradient-Based Meta-Learning for Enhanced Meta-Gradient Estimation
This work addresses the meta-gradient approximation error in gradient-based meta-learning, offering a more accurate and efficient method for few-shot learning tasks.
Binomial gradient-based meta-learning (BinomGBML) uses a truncated binomial expansion for meta-gradient estimation, improving accuracy over truncated backpropagation. Applied to MAML, it achieves provably super-exponential error decay and boosted performance with slightly increased computational overhead.
Meta-learning offers a principled framework leveraging \emph{task-invariant} priors from related tasks, with which \emph{task-specific} models can be fine-tuned on downstream tasks, even with limited data records. Gradient-based meta-learning (GBML) relies on gradient descent (GD) to adapt the prior to a new task. Albeit effective, these methods incur high computational overhead that scales linearly with the number of GD steps. To enhance efficiency and scalability, existing methods approximate the gradient of prior parameters (meta-gradient) via truncated backpropagation, yet suffer large approximation errors. Targeting accurate approximation, this work puts forth binomial GBML (BinomGBML), which relies on a truncated binomial expansion for meta-gradient estimation. This novel expansion endows more information in the meta-gradient estimation via efficient parallel computation. As a running paradigm applied to model-agnostic meta-learning (MAML), the resultant BinomMAML provably enjoys error bounds that not only improve upon existing approaches, but also decay super-exponentially under mild conditions. Numerical tests corroborate the theoretical analysis and showcase boosted performance with slightly increased computational overhead.