Privacy-Preserving Reinforcement Learning from Human Feedback via Decoupled Reward Modeling
This addresses privacy concerns in fine-tuning large language models with human feedback, which is an incremental improvement over existing methods.
The paper tackles the problem of protecting sensitive user data in reinforcement learning from human feedback by proposing a differentially private framework that applies privacy only to reward learning, showing theoretically that privacy adds an additive error term and empirically achieving stronger alignment performance than baselines on the Anthropic HH-RLHF dataset with Gemma-2B-IT.
Preference-based fine-tuning has become an important component in training large language models, and the data used at this stage may contain sensitive user information. A central question is how to design a differentially private pipeline that is well suited to the distinct structure of reinforcement learning from human feedback. We propose a privacy-preserving framework that imposes differential privacy only on reward learning and derives the final policy from the resulting private reward model. Theoretically, we study the suboptimality gap and show that privacy contributes an additional additive term beyond the usual non-private statistical error. We also establish a minimax lower bound and show that the dominant term changes with sample size and privacy level, which in turn characterizes regimes in which the upper bound is rate-optimal up to logarithmic factors. Empirically, synthetic experiments confirm the scaling predicted by the theory, and experiments on the Anthropic HH-RLHF dataset using the Gemma-2B-IT model show stronger private alignment performance than existing differentially private baseline methods across privacy budgets.