TextNAS: A Neural Architecture Search Space tailored for Text Representation
This work addresses the challenge of designing effective text representation networks for natural language processing tasks, offering a domain-specific improvement through a novel search space.
The authors tackled the problem of finding optimal text representation networks by proposing a tailored Neural Architecture Search (NAS) search space, which discovered architectures that outperformed state-of-the-art models on text classification and natural language inference tasks across various public datasets.
Learning text representation is crucial for text classification and other language related tasks. There are a diverse set of text representation networks in the literature, and how to find the optimal one is a non-trivial problem. Recently, the emerging Neural Architecture Search (NAS) techniques have demonstrated good potential to solve the problem. Nevertheless, most of the existing works of NAS focus on the search algorithms and pay little attention to the search space. In this paper, we argue that the search space is also an important human prior to the success of NAS in different applications. Thus, we propose a novel search space tailored for text representation. Through automatic search, the discovered network architecture outperforms state-of-the-art models on various public datasets on text classification and natural language inference tasks. Furthermore, some of the design principles found in the automatic network agree well with human intuition.