The Efficiency Misnomer
This work addresses a reporting issue for researchers and practitioners in ML, but it is incremental as it focuses on improving existing practices rather than introducing new methods.
The paper tackles the problem of incomplete reporting of model efficiency metrics in machine learning, demonstrating that common cost indicators can contradict each other and lead to partial conclusions about practical model considerations.
Model efficiency is a critical aspect of developing and deploying machine learning models. Inference time and latency directly affect the user experience, and some applications have hard requirements. In addition to inference costs, model training also have direct financial and environmental impacts. Although there are numerous well-established metrics (cost indicators) for measuring model efficiency, researchers and practitioners often assume that these metrics are correlated with each other and report only few of them. In this paper, we thoroughly discuss common cost indicators, their advantages and disadvantages, and how they can contradict each other. We demonstrate how incomplete reporting of cost indicators can lead to partial conclusions and a blurred or incomplete picture of the practical considerations of different models. We further present suggestions to improve reporting of efficiency metrics.