Some things are more CRINGE than others: Iterative Preference Optimization with the Pairwise Cringe Loss
This work addresses the challenge of efficiently training models with pairwise preference data for practitioners in AI alignment, though it is incremental as it builds on an existing binary feedback method.
The authors tackled the problem of aligning large language models with pairwise preferences by extending the binary feedback Cringe Loss to a pairwise setting, achieving state-of-the-art performance on the AlpacaFarm benchmark compared to methods like PPO and DPO.
Practitioners commonly align large language models using pairwise preferences, i.e., given labels of the type response A is preferred to response B for a given input. Perhaps less commonly, methods have also been developed for binary feedback, i.e. training models given labels of type response A is good or bad. We show how an existing performant binary feedback method, the Cringe Loss (Adolphs et al., 2022), can be generalized to the pairwise preference setting using a simple soft margin extension. Pairwise Cringe Loss is straightforward to implement and efficient to train, and we find it outperforms state-of-the-art preference optimization algorithms such as PPO and DPO on the AlpacaFarm benchmark. We show that iterations of training of our model are important for improved results, and that we can generalize DPO to Iterative DPO in the same way.