Beyond Surrogates: A Quantitative Analysis for Inter-Metric Relationships
This work aims to help practitioners design evaluation systems that align offline improvements with online objectives, addressing a common problem in industrial applications.
This paper addresses the "Metric Mismatch" problem where offline metric gains do not translate to online performance by proposing a unified theoretical framework to quantify relationships between different evaluation metrics. It categorizes metrics and analyzes their relationships through Bayes-Optimal Set and Regret Transfer, identifying structural asymmetry in regret transfer.
The Consistency property between surrogate losses and evaluation metrics has been extensively studied to ensure that minimizing a loss leads to metric optimality. However, the direct relationship between different evaluation metrics remains significantly underexplored. This theoretical gap results in the "Metric Mismatch" frequently observed in industrial applications, where gains in offline validation metrics fail to translate into online performance. To bridge this disconnection, this paper proposes a unified theoretical framework designed to quantify the relationships between metrics. We categorize metrics into different classes to facilitate a comparative analysis across different mathematical forms and interrogates these relationships through Bayes-Optimal Set and Regret Transfer. Through this framework, we provide a new perspective on identifying the structural asymmetry in regret transfer, enabling the design of evaluation systems that are theoretically guaranteed to align offline improvements with online objectives.