Towards Understanding Generalization in Gradient-Based Meta-Learning
This work addresses generalization challenges in meta-learning for researchers, offering insights into landscape properties and a new regularization method, though it is incremental in building on existing MAML frameworks.
The paper investigates generalization in gradient-based meta-learning by analyzing objective landscapes, finding that meta-test solutions become flatter and more distant from meta-train solutions as training progresses, even when generalization degrades, challenging the link between flat minima and generalization. It proposes a novel regularizer for MAML based on coherence in adaptation trajectories and gradients, showing experimental effectiveness.
In this work we study generalization of neural networks in gradient-based meta-learning by analyzing various properties of the objective landscapes. We experimentally demonstrate that as meta-training progresses, the meta-test solutions, obtained after adapting the meta-train solution of the model, to new tasks via few steps of gradient-based fine-tuning, become flatter, lower in loss, and further away from the meta-train solution. We also show that those meta-test solutions become flatter even as generalization starts to degrade, thus providing an experimental evidence against the correlation between generalization and flat minima in the paradigm of gradient-based meta-leaning. Furthermore, we provide empirical evidence that generalization to new tasks is correlated with the coherence between their adaptation trajectories in parameter space, measured by the average cosine similarity between task-specific trajectory directions, starting from a same meta-train solution. We also show that coherence of meta-test gradients, measured by the average inner product between the task-specific gradient vectors evaluated at meta-train solution, is also correlated with generalization. Based on these observations, we propose a novel regularizer for MAML and provide experimental evidence for its effectiveness.