L3Ms -- Lagrange Large Language Models
This addresses the challenge of customizing LLM alignment for diverse applications, though it appears incremental as it builds on existing optimization techniques.
The paper tackles the problem of supervised fine-tuning and alignment of large language models by formulating it as a constrained optimization problem, proposing L3Ms with logarithmic barriers to enforce application-specific requirements without heuristics, and demonstrates their versatility and efficacy in tailored alignments.
Supervised fine-tuning (SFT) and alignment of large language models (LLMs) are key steps in providing a good user experience. However, the concept of an appropriate alignment is inherently application-dependent, and current methods often rely on heuristic choices to drive optimization. In this work, we formulate SFT and alignment as a constrained optimization problem: the LLM is fine-tuned on a task while being required to meet application-specific requirements, without resorting to heuristics. To solve this, we propose Lagrange Large Language Models (L3Ms), which employ logarithmic barriers to enforce the constraints. This approach allows for the customization of L3Ms across diverse applications while avoiding heuristic-driven processes. We experimentally demonstrate the versatility and efficacy of L3Ms in achieving tailored alignments for various applications.