Multi-Modal Fusion by Meta-Initialization
This work addresses few-shot learning for AI systems by improving adaptation with multi-modal data, though it is incremental as it builds on MAML.
The authors tackled the problem of few-shot learning by extending Model-Agnostic Meta-Learning (MAML) to incorporate auxiliary textual information, resulting in FuMI, which significantly outperforms uni-modal baselines like MAML on the new iNat-Anim dataset.
When experience is scarce, models may have insufficient information to adapt to a new task. In this case, auxiliary information - such as a textual description of the task - can enable improved task inference and adaptation. In this work, we propose an extension to the Model-Agnostic Meta-Learning algorithm (MAML), which allows the model to adapt using auxiliary information as well as task experience. Our method, Fusion by Meta-Initialization (FuMI), conditions the model initialization on auxiliary information using a hypernetwork, rather than learning a single, task-agnostic initialization. Furthermore, motivated by the shortcomings of existing multi-modal few-shot learning benchmarks, we constructed iNat-Anim - a large-scale image classification dataset with succinct and visually pertinent textual class descriptions. On iNat-Anim, FuMI significantly outperforms uni-modal baselines such as MAML in the few-shot regime. The code for this project and a dataset exploration tool for iNat-Anim are publicly available at https://github.com/s-a-malik/multi-few .