A Gradient Analysis Framework for Rewarding Good and Penalizing Bad Examples in Language Models
This provides a unified framework for optimizing language models, though it is incremental as it builds on existing methods like DPO and ExMATE.
The paper tackles the problem of improving language model output quality by systematically comparing methods that reward good examples and penalize bad ones, finding that combining DPO with ExMATE instead of MLE enhances statistical performance by 5-7% and generative win rate by +18%.
Beyond maximum likelihood estimation (MLE), the standard objective of a language model (LM) that optimizes good examples probabilities, many studies have explored ways that also penalize bad examples for enhancing the quality of output distribution, including unlikelihood training, exponential maximizing average treatment effect (ExMATE), and direct preference optimization (DPO). To systematically compare these methods and further provide a unified recipe for LM optimization, in this paper, we present a unique angle of gradient analysis of loss functions that simultaneously reward good examples and penalize bad ones in LMs. Through both mathematical results and experiments on CausalDialogue and Anthropic HH-RLHF datasets, we identify distinct functional characteristics among these methods. We find that ExMATE serves as a superior surrogate for MLE, and that combining DPO with ExMATE instead of MLE further enhances both the statistical (5-7%) and generative (+18% win rate) performance.