ReSURE: Regularizing Supervision Unreliability for Multi-turn Dialogue Fine-tuning
This addresses the challenge of effectively utilizing large-scale, potentially noisy data for fine-tuning dialogue systems, though it is incremental as it builds on existing methods for handling data quality.
The paper tackles the problem of performance degradation in multi-turn dialogue fine-tuning due to low-quality supervision and error propagation, proposing ReSURE to dynamically down-weight unreliable supervision, which improves stability and response quality with positive Spearman correlations (0.21 to 1.0) across benchmarks.
Fine-tuning multi-turn dialogue systems requires high-quality supervision but often suffers from degraded performance when exposed to low-quality data. Supervision errors in early turns can propagate across subsequent turns, undermining coherence and response quality. Existing methods typically address data quality via static prefiltering, which decouples quality control from training and fails to mitigate turn-level error propagation. In this context, we propose ReSURE (Regularizing Supervision UnREliability), an adaptive learning method that dynamically down-weights unreliable supervision without explicit filtering. ReSURE estimates per-turn loss distributions using Welford's online statistics and reweights sample losses on the fly accordingly. Experiments on both single-source and mixed-quality datasets show improved stability and response quality. Notably, ReSURE enjoys positive Spearman correlations (0.21 ~ 1.0 across multiple benchmarks) between response scores and number of samples regardless of data quality, which potentially paves the way for utilizing large-scale data effectively. Code is publicly available at https://github.com/Elvin-Yiming-Du/ReSURE_Multi_Turn_Training.