Calibration-then-Calculation: A Variance Reduced Metric Framework in Deep Click-Through Rate Prediction Models
This addresses the issue of unreliable performance comparisons in deep learning for researchers and practitioners, though it is incremental as it builds on existing metric frameworks.
The paper tackles the problem of high variance in neural network evaluation metrics due to training randomness, which leads to imprecise comparisons in deep learning pipelines. It introduces the Calibrated Loss Metric, a novel framework that reduces variance and improves accuracy in detecting modeling improvements, with experimental validations in Deep Click-Through Rate Prediction Models.
The adoption of deep learning across various fields has been extensive, yet there is a lack of focus on evaluating the performance of deep learning pipelines. Typically, with the increased use of large datasets and complex models, the training process is run only once and the result is compared to previous benchmarks. This practice can lead to imprecise comparisons due to the variance in neural network evaluation metrics, which stems from the inherent randomness in the training process. Traditional solutions, such as running the training process multiple times, are often infeasible due to computational constraints. In this paper, we introduce a novel metric framework, the Calibrated Loss Metric, designed to address this issue by reducing the variance present in its conventional counterpart. Consequently, this new metric enhances the accuracy in detecting effective modeling improvements. Our approach is substantiated by theoretical justifications and extensive experimental validations within the context of Deep Click-Through Rate Prediction Models.