Interpretable Reward Modeling with Active Concept Bottlenecks
This work addresses the need for more transparent and auditable reward models in AI alignment, representing an incremental advancement in interpretable preference learning.
The paper tackled the problem of opaque reward functions in reinforcement learning from human feedback by introducing a framework that decomposes reward prediction into interpretable concepts, achieving improved interpretability and sample efficiency on the UltraFeedback dataset.
We introduce Concept Bottleneck Reward Models (CB-RM), a reward modeling framework that enables interpretable preference learning through selective concept annotation. Unlike standard RLHF methods that rely on opaque reward functions, CB-RM decomposes reward prediction into human-interpretable concepts. To make this framework efficient in low-supervision settings, we formalize an active learning strategy that dynamically acquires the most informative concept labels. We propose an acquisition function based on Expected Information Gain and show that it significantly accelerates concept learning without compromising preference accuracy. Evaluated on the UltraFeedback dataset, our method outperforms baselines in interpretability and sample efficiency, marking a step towards more transparent, auditable, and human-aligned reward models.