On the Value of ML Models
This addresses the problem of misaligned evaluation metrics in ML research for practitioners, though it is incremental as it focuses on a specific use case.
The paper argues that ML models should be evaluated using metrics that better capture their practical value, and demonstrates for selective classification that this approach is simple, has important consequences, and provides insights into what makes a good model.
We argue that, when establishing and benchmarking Machine Learning (ML) models, the research community should favour evaluation metrics that better capture the value delivered by their model in practical applications. For a specific class of use cases -- selective classification -- we show that not only can it be simple enough to do, but that it has import consequences and provides insights what to look for in a ``good'' ML model.