Configurable Preference Tuning with Rubric-Guided Synthetic Data
This addresses the problem of inflexible AI alignment for users needing adaptable models, representing a novel method rather than an incremental improvement.
The paper tackles the limitation of static preferences in AI alignment models by introducing Configurable Preference Tuning (CPT), a framework that enables language models to dynamically adjust behavior based on explicit directives, achieving fine-grained control without retraining.
Models of human feedback for AI alignment, such as those underpinning Direct Preference Optimization (DPO), often bake in a singular, static set of preferences, limiting adaptability. This paper challenges the assumption of monolithic preferences by introducing Configurable Preference Tuning (CPT), a novel framework for endowing language models with the ability to dynamically adjust their behavior based on explicit, human-interpretable directives. CPT leverages synthetically generated preference data, conditioned on system prompts derived from structured, fine-grained rubrics that define desired attributes like writing style. By fine-tuning with these rubric-guided preferences, the LLM learns to modulate its outputs at inference time in response to the system prompt, without retraining. This approach not only offers fine-grained control but also provides a mechanism for modeling more nuanced and context-dependent human feedback. Several experimental artifacts, such as training code, generated datasets and fine-tuned models are released at https://github.com/vicgalle/configurable-preference-tuning