Learning a Canonical Basis of Human Preferences from Binary Ratings
This work addresses the need for interpretable and efficient human preference modeling in AI alignment, offering a method to generalize preferences across datasets and topics, though it is incremental in building on existing RLHF techniques.
The paper tackled the problem of understanding human preferences in alignment datasets, finding that 21 preference categories capture over 89% of variation across individuals, analogous to a canonical basis. It demonstrated this basis's utility in model evaluation and training, showing fine-tuning on subsets aligns models accordingly.
Recent advances in generative AI have been driven by alignment techniques such as reinforcement learning from human feedback (RLHF). RLHF and related techniques typically involve constructing a dataset of binary or ranked choice human preferences and subsequently fine-tuning models to align with these preferences. This paper shifts the focus to understanding the preferences encoded in such datasets and identifying common human preferences. We find that a small subset of 21 preference categories (selected from a set of nearly 5,000 distinct preferences) captures >89% of preference variation across individuals. This small set of preferences is analogous to a canonical basis of human preferences, similar to established findings that characterize human variation in psychology or facial recognition studies. Through both synthetic and empirical evaluations, we confirm that our low-rank, canonical set of human preferences generalizes across the entire dataset and within specific topics. We further demonstrate our preference basis' utility in model evaluation, where our preference categories offer deeper insights into model alignment, and in model training, where we show that fine-tuning on preference-defined subsets successfully aligns the model accordingly.