Pragmatic Feature Preferences: Learning Reward-Relevant Preferences from Human Input
This work addresses the challenge of aligning AI systems with human intentions by improving reward learning efficiency, though it is incremental as it builds on existing preference-based methods.
The paper tackles the problem of learning reward models from human preferences by incorporating feature-level preferences alongside example comparisons, resulting in more efficient convergence to accurate rewards with fewer comparisons in linear bandit settings and a real-world mushroom foraging task.
Humans use social context to specify preferences over behaviors, i.e. their reward functions. Yet, algorithms for inferring reward models from preference data do not take this social learning view into account. Inspired by pragmatic human communication, we study how to extract fine-grained data regarding why an example is preferred that is useful for learning more accurate reward models. We propose to enrich binary preference queries to ask both (1) which features of a given example are preferable in addition to (2) comparisons between examples themselves. We derive an approach for learning from these feature-level preferences, both for cases where users specify which features are reward-relevant, and when users do not. We evaluate our approach on linear bandit settings in both vision- and language-based domains. Results support the efficiency of our approach in quickly converging to accurate rewards with fewer comparisons vs. example-only labels. Finally, we validate the real-world applicability with a behavioral experiment on a mushroom foraging task. Our findings suggest that incorporating pragmatic feature preferences is a promising approach for more efficient user-aligned reward learning.