Benchmarking Long-tail Generalization with Likelihood Splits
This addresses the need for more robust benchmarks in NLP to test long-tail generalization, though it is incremental as it builds on existing datasets and methods.
The authors tackled the problem of evaluating NLP systems' generalization to rare utterances by proposing Likelihood Splits, a method to create challenging benchmarks from existing datasets, which increased error rates by up to 93% compared to random splits.
In order to reliably process natural language, NLP systems must generalize to the long tail of rare utterances. We propose a method to create challenging benchmarks that require generalizing to the tail of the distribution by re-splitting existing datasets. We create 'Likelihood Splits' where examples that are assigned lower likelihood by a pre-trained language model (LM) are placed in the test set, and more likely examples are in the training set. This simple approach can be customized to construct meaningful train-test splits for a wide range of tasks. Likelihood Splits surface more challenges than random splits: relative error rates of state-of-the-art models increase by 59% for semantic parsing on Spider, 93% for natural language inference on SNLI, and 33% for yes/no question answering on BoolQ, on our splits compared with the corresponding random splits. Moreover, Likelihood Splits create fairer benchmarks than adversarial filtering; when the LM used to create the splits is also employed as the task model, our splits do not unfairly penalize the LM.