NAS-Bench-NLP: Neural Architecture Search Benchmark for Natural Language Processing
This provides a benchmark for NAS in NLP, enabling more reproducible and accessible research for those with limited computational resources, though it is incremental as it extends existing NAS benchmark concepts to a new domain.
The authors tackled the lack of neural architecture search (NAS) benchmarks for natural language processing by creating NAS-Bench-NLP, which includes a search space of recurrent neural networks trained on 14k architectures and evaluated on semantic and language understanding tasks. They demonstrated its utility by testing several NAS algorithms.
Neural Architecture Search (NAS) is a promising and rapidly evolving research area. Training a large number of neural networks requires an exceptional amount of computational power, which makes NAS unreachable for those researchers who have limited or no access to high-performance clusters and supercomputers. A few benchmarks with precomputed neural architectures performances have been recently introduced to overcome this problem and ensure more reproducible experiments. However, these benchmarks are only for the computer vision domain and, thus, are built from the image datasets and convolution-derived architectures. In this work, we step outside the computer vision domain by leveraging the language modeling task, which is the core of natural language processing (NLP). Our main contribution is as follows: we have provided search space of recurrent neural networks on the text datasets and trained 14k architectures within it; we have conducted both intrinsic and extrinsic evaluation of the trained models using datasets for semantic relatedness and language understanding evaluation; finally, we have tested several NAS algorithms to demonstrate how the precomputed results can be utilized. We believe that our results have high potential of usage for both NAS and NLP communities.