Transductive Learning for Textual Few-Shot Classification in API-based Embedding Models
This addresses practical challenges in NLP for users of proprietary APIs, offering a privacy-preserving and efficient solution, though it is incremental in applying transductive learning to a specific scenario.
The paper tackles few-shot classification with API-based embedding models under compute-cost and privacy constraints by proposing a transductive inference method with a Fisher-Rao loss regularizer, showing superiority over inductive methods in evaluations across eight datasets and 1,000 episodes.
Proprietary and closed APIs are becoming increasingly common to process natural language, and are impacting the practical applications of natural language processing, including few-shot classification. Few-shot classification involves training a model to perform a new classification task with a handful of labeled data. This paper presents three contributions. First, we introduce a scenario where the embedding of a pre-trained model is served through a gated API with compute-cost and data-privacy constraints. Second, we propose a transductive inference, a learning paradigm that has been overlooked by the NLP community. Transductive inference, unlike traditional inductive learning, leverages the statistics of unlabeled data. We also introduce a new parameter-free transductive regularizer based on the Fisher-Rao loss, which can be used on top of the gated API embeddings. This method fully utilizes unlabeled data, does not share any label with the third-party API provider and could serve as a baseline for future research. Third, we propose an improved experimental setting and compile a benchmark of eight datasets involving multiclass classification in four different languages, with up to 151 classes. We evaluate our methods using eight backbone models, along with an episodic evaluation over 1,000 episodes, which demonstrate the superiority of transductive inference over the standard inductive setting.