Many of Your DPOs are Secretly One: Attempting Unification Through Mutual Information
This provides a clearer, structured approach for researchers navigating DPO variants, though it is incremental as it unifies existing methods rather than introducing a new paradigm.
The paper tackles the proliferation of Direct Preference Optimization (DPO) variants for aligning large language models by proposing a unifying framework based on mutual information with flexible priors, showing that many existing algorithms like SimPO, TDPO, and SparsePO can be derived from it.
Post-alignment of large language models (LLMs) is critical in improving their utility, safety, and alignment with human intentions. Direct preference optimisation (DPO) has become one of the most widely used algorithms for achieving this alignment, given its ability to optimise models based on human feedback directly. However, the vast number of DPO variants in the literature has made it increasingly difficult for researchers to navigate and fully grasp the connections between these approaches. This paper introduces a unifying framework inspired by mutual information, which proposes a new loss function with flexible priors. By carefully specifying these priors, we demonstrate that many existing algorithms, such as SimPO, TDPO, SparsePO, and others, can be derived from our framework. This unification offers a clearer and more structured approach, allowing researchers to understand the relationships between different DPO variants better. We aim to simplify the landscape of DPO algorithms, making it easier for the research community to gain insights and foster further advancements in LLM alignment. Ultimately, we hope our framework can be a foundation for developing more robust and interpretable alignment techniques.