HULK: An Energy Efficiency Benchmark Platform for Responsible Natural Language Processing
This addresses energy consumption concerns for NLP practitioners and researchers, though it is incremental as it builds on existing benchmarking efforts.
The authors tackled the problem of energy efficiency in pretrained NLP models by introducing HULK, a benchmarking platform that compares models' efficiency in terms of time and cost, revealing that fine-tuning efficiency varies significantly across tasks and parameter count does not guarantee better efficiency.
Computation-intensive pretrained models have been taking the lead of many natural language processing benchmarks such as GLUE. However, energy efficiency in the process of model training and inference becomes a critical bottleneck. We introduce HULK, a multi-task energy efficiency benchmarking platform for responsible natural language processing. With HULK, we compare pretrained models' energy efficiency from the perspectives of time and cost. Baseline benchmarking results are provided for further analysis. The fine-tuning efficiency of different pretrained models can differ a lot among different tasks and fewer parameter number does not necessarily imply better efficiency. We analyzed such phenomenon and demonstrate the method of comparing the multi-task efficiency of pretrained models. Our platform is available at https://sites.engineering.ucsb.edu/~xiyou/hulk/.