FewNLU: Benchmarking State-of-the-Art Methods for Few-Shot Natural Language Understanding
This work addresses the issue of fair comparison in few-shot NLU for researchers, but it is incremental as it focuses on evaluation rather than new methods.
The authors tackled the problem of inconsistent evaluation protocols in few-shot natural language understanding by introducing a new framework that improves test performance, dev-test correlation, and stability, revealing that prior methods' performance was inaccurately estimated and no single method dominates most tasks.
The few-shot natural language understanding (NLU) task has attracted much recent attention. However, prior methods have been evaluated under a disparate set of protocols, which hinders fair comparison and measuring progress of the field. To address this issue, we introduce an evaluation framework that improves previous evaluation procedures in three key aspects, i.e., test performance, dev-test correlation, and stability. Under this new evaluation framework, we re-evaluate several state-of-the-art few-shot methods for NLU tasks. Our framework reveals new insights: (1) both the absolute performance and relative gap of the methods were not accurately estimated in prior literature; (2) no single method dominates most tasks with consistent performance; (3) improvements of some methods diminish with a larger pretrained model; and (4) gains from different methods are often complementary and the best combined model performs close to a strong fully-supervised baseline. We open-source our toolkit, FewNLU, that implements our evaluation framework along with a number of state-of-the-art methods.