MOROCCO: Model Resource Comparison Framework
This addresses the resource efficiency challenge for NLP practitioners deploying models in real-world environments, but it is incremental as it builds on existing evaluation frameworks.
The paper tackles the problem of high computational costs of pre-trained NLP models in production by introducing MOROCCO, a framework that evaluates models based on quality metrics, memory footprint, and inference time, demonstrating its applicability on GLUE-like suites in multiple languages.
The new generation of pre-trained NLP models push the SOTA to the new limits, but at the cost of computational resources, to the point that their use in real production environments is often prohibitively expensive. We tackle this problem by evaluating not only the standard quality metrics on downstream tasks but also the memory footprint and inference time. We present MOROCCO, a framework to compare language models compatible with \texttt{jiant} environment which supports over 50 NLU tasks, including SuperGLUE benchmark and multiple probing suites. We demonstrate its applicability for two GLUE-like suites in different languages.