Unsupervised Ranking and Aggregation of Label Descriptions for Zero-Shot Classifiers
This work addresses a key bottleneck in zero-shot classification for NLP practitioners by enabling better label description selection in true zero-shot setups, though it is incremental as it builds on existing probabilistic models.
The paper tackles the challenge of designing effective label descriptions for zero-shot text classifiers without a development set, proposing an unsupervised method that uses probabilistic models from repeated rating analysis to select and aggregate multiple noisy label descriptions, achieving performance improvements across sentiment, topic, and stance tasks.
Zero-shot text classifiers based on label descriptions embed an input text and a set of labels into the same space: measures such as cosine similarity can then be used to select the most similar label description to the input text as the predicted label. In a true zero-shot setup, designing good label descriptions is challenging because no development set is available. Inspired by the literature on Learning with Disagreements, we look at how probabilistic models of repeated rating analysis can be used for selecting the best label descriptions in an unsupervised fashion. We evaluate our method on a set of diverse datasets and tasks (sentiment, topic and stance). Furthermore, we show that multiple, noisy label descriptions can be aggregated to boost the performance.