Shaping Visual Representations with Language for Few-shot Classification
This work addresses a specific scenario in few-shot learning for computer vision, offering a simpler and more data-efficient approach, though it is incremental as it builds on existing methods for language-guided classification.
The paper tackled the problem of few-shot visual classification when natural language task descriptions are available during training but not at test time, by proposing language-shaped learning (LSL) to regularize visual representations to predict language, resulting in improved performance over baselines in two challenging domains.
By describing the features and abstractions of our world, language is a crucial tool for human learning and a promising source of supervision for machine learning models. We use language to improve few-shot visual classification in the underexplored scenario where natural language task descriptions are available during training, but unavailable for novel tasks at test time. Existing models for this setting sample new descriptions at test time and use those to classify images. Instead, we propose language-shaped learning (LSL), an end-to-end model that regularizes visual representations to predict language. LSL is conceptually simpler, more data efficient, and outperforms baselines in two challenging few-shot domains.