TRIAGE: Characterizing and auditing training data for improved regression
This work addresses the need for better training data analysis in regression applications, offering a novel method that can enhance model performance and enable new approaches to dataset selection and feature acquisition.
The paper tackles the problem of data characterization for regression tasks, which is understudied compared to classification, by introducing TRIAGE, a framework that uses conformal predictive distributions to score and characterize training data, showing its utility for improving performance through data sculpting and filtering in multiple regression settings.
Data quality is crucial for robust machine learning algorithms, with the recent interest in data-centric AI emphasizing the importance of training data characterization. However, current data characterization methods are largely focused on classification settings, with regression settings largely understudied. To address this, we introduce TRIAGE, a novel data characterization framework tailored to regression tasks and compatible with a broad class of regressors. TRIAGE utilizes conformal predictive distributions to provide a model-agnostic scoring method, the TRIAGE score. We operationalize the score to analyze individual samples' training dynamics and characterize samples as under-, over-, or well-estimated by the model. We show that TRIAGE's characterization is consistent and highlight its utility to improve performance via data sculpting/filtering, in multiple regression settings. Additionally, beyond sample level, we show TRIAGE enables new approaches to dataset selection and feature acquisition. Overall, TRIAGE highlights the value unlocked by data characterization in real-world regression applications