Function Contrastive Learning of Transferable Meta-Representations
This work addresses the challenge of improving meta-learning for fast adaptation in tasks sharing underlying functions, though it appears incremental as it builds on existing supervised meta-learning frameworks.
The paper tackles the problem of learning transferable meta-representations that are robust to noise and generalize across downstream tasks by proposing a decoupled encoder-decoder approach with contrastive self-supervision. The result shows that their method outperforms strong baselines in downstream performance and noise robustness on synthetic and real-world datasets.
Meta-learning algorithms adapt quickly to new tasks that are drawn from the same task distribution as the training tasks. The mechanism leading to fast adaptation is the conditioning of a downstream predictive model on the inferred representation of the task's underlying data generative process, or \emph{function}. This \emph{meta-representation}, which is computed from a few observed examples of the underlying function, is learned jointly with the predictive model. In this work, we study the implications of this joint training on the transferability of the meta-representations. Our goal is to learn meta-representations that are robust to noise in the data and facilitate solving a wide range of downstream tasks that share the same underlying functions. To this end, we propose a decoupled encoder-decoder approach to supervised meta-learning, where the encoder is trained with a contrastive objective to find a good representation of the underlying function. In particular, our training scheme is driven by the self-supervision signal indicating whether two sets of examples stem from the same function. Our experiments on a number of synthetic and real-world datasets show that the representations we obtain outperform strong baselines in terms of downstream performance and noise robustness, even when these baselines are trained in an end-to-end manner.