Wormhole MAML: Meta-Learning in Glued Parameter Space
This addresses a specific bottleneck in meta-learning for researchers, though it appears incremental as a variation of existing MAML methods.
The paper tackles the problem of conflicting gradients in model-agnostic meta-learning by introducing a multiplicative parameter in the inner-loop adaptation, which creates shortcuts in parameter space and improves training dynamics. The method shows theoretical and numerical improvements, with experiments on toy classification, regression, and MNIST classification problems.
In this paper, we introduce a novel variation of model-agnostic meta-learning, where an extra multiplicative parameter is introduced in the inner-loop adaptation. Our variation creates a shortcut in the parameter space for the inner-loop adaptation and increases model expressivity in a highly controllable manner. We show both theoretically and numerically that our variation alleviates the problem of conflicting gradients and improves training dynamics. We conduct experiments on 3 distinctive problems, including a toy classification problem for threshold comparison, a regression problem for wavelet transform, and a classification problem on MNIST. We also discuss ways to generalize our method to a broader class of problems.