Few-shot Sequence Learning with Transformers
This work addresses the problem of efficient few-shot learning for sequence data, which is relevant for researchers and practitioners working with limited labeled data in sequence-based tasks.
This paper explores few-shot learning for sequence data using Transformers. The authors propose an efficient method where a task-specific token is appended to the input sequence, and its embedding is optimized with a few labeled examples, achieving performance comparable to other methods while being more computationally efficient.
Few-shot algorithms aim at learning new tasks provided only a handful of training examples. In this work we investigate few-shot learning in the setting where the data points are sequences of tokens and propose an efficient learning algorithm based on Transformers. In the simplest setting, we append a token to an input sequence which represents the particular task to be undertaken, and show that the embedding of this token can be optimized on the fly given few labeled examples. Our approach does not require complicated changes to the model architecture such as adapter layers nor computing second order derivatives as is currently popular in the meta-learning and few-shot learning literature. We demonstrate our approach on a variety of tasks, and analyze the generalization properties of several model variants and baseline approaches. In particular, we show that compositional task descriptors can improve performance. Experiments show that our approach works at least as well as other methods, while being more computationally efficient.