W-PCA Based Gradient-Free Proxy for Efficient Search of Lightweight Language Models
This work addresses the need for efficient NLP systems by providing a faster method for searching lightweight models, though it is incremental as it builds on zero-shot NAS approaches.
The paper tackles the problem of efficiently designing lightweight language models by introducing W-PCA, a zero-shot neural architecture search method that uses parameter count and principal components in feed-forward layers as proxies, eliminating gradient computations. The results show it significantly reduces training time compared to one-shot NAS and achieves higher scores on GLUE and SQuAD datasets than previous training-based methods.
The demand for efficient natural language processing (NLP) systems has led to the development of lightweight language models. Previous work in this area has primarily focused on manual design or training-based neural architecture search (NAS) methods. Recently, zero-shot NAS methods have been proposed for evaluating language models without the need for training. However, prevailing approaches to zero-shot NAS often face challenges such as biased evaluation metrics and computational inefficiencies. In this paper, we introduce weight-weighted PCA (W-PCA), a novel zero-shot NAS method specifically tailored for lightweight language models. Our approach utilizes two evaluation proxies: the parameter count and the number of principal components with cumulative contribution exceeding $η$ in the feed-forward neural (FFN) layer. Additionally, by eliminating the need for gradient computations, we optimize the evaluation time, thus enhancing the efficiency of designing and evaluating lightweight language models. We conduct a comparative analysis on the GLUE and SQuAD datasets to evaluate our approach. The results demonstrate that our method significantly reduces training time compared to one-shot NAS methods and achieves higher scores in the testing phase compared to previous state-of-the-art training-based methods. Furthermore, we perform ranking evaluations on a dataset sampled from the FlexiBERT search space. Our approach exhibits superior ranking correlation and further reduces solving time compared to other zero-shot NAS methods that require gradient computation.