Distilling Neural Networks for Greener and Faster Dependency Parsing
This work addresses efficiency and environmental concerns for NLP researchers and practitioners, though it is incremental as it applies an existing compression technique to a specific parser.
The paper tackled the problem of high carbon footprint in NLP by using teacher-student distillation to compress the Biaffine dependency parser, achieving a 2.30x speedup on CPU with only a ~1 point drop in accuracy when reduced to 20% of parameters, and even outperforming the fastest modern parser on the Penn Treebank in some cases.
The carbon footprint of natural language processing research has been increasing in recent years due to its reliance on large and inefficient neural network implementations. Distillation is a network compression technique which attempts to impart knowledge from a large model to a smaller one. We use teacher-student distillation to improve the efficiency of the Biaffine dependency parser which obtains state-of-the-art performance with respect to accuracy and parsing speed (Dozat and Manning, 2017). When distilling to 20\% of the original model's trainable parameters, we only observe an average decrease of $\sim$1 point for both UAS and LAS across a number of diverse Universal Dependency treebanks while being 2.30x (1.19x) faster than the baseline model on CPU (GPU) at inference time. We also observe a small increase in performance when compressing to 80\% for some treebanks. Finally, through distillation we attain a parser which is not only faster but also more accurate than the fastest modern parser on the Penn Treebank.