Datasets: A Community Library for Natural Language Processing
This provides a standardized tool for NLP researchers to access and manage datasets, though it is incremental as it builds on existing data-sharing practices.
The authors tackled the problem of managing the rapidly growing scale and variety of NLP datasets by developing Datasets, a community library that standardizes interfaces and documentation. After a year, it includes over 650 datasets, has 250+ contributors, and supports cross-dataset research.
The scale, variety, and quantity of publicly-available NLP datasets has grown rapidly as researchers propose new tasks, larger models, and novel benchmarks. Datasets is a community library for contemporary NLP designed to support this ecosystem. Datasets aims to standardize end-user interfaces, versioning, and documentation, while providing a lightweight front-end that behaves similarly for small datasets as for internet-scale corpora. The design of the library incorporates a distributed, community-driven approach to adding datasets and documenting usage. After a year of development, the library now includes more than 650 unique datasets, has more than 250 contributors, and has helped support a variety of novel cross-dataset research projects and shared tasks. The library is available at https://github.com/huggingface/datasets.