Towards User-level Private Reinforcement Learning with Human Feedback
This work addresses a crucial privacy issue for users in RLHF applications, offering a more effective solution than previous item-level approaches, though it is incremental in advancing privacy techniques within this specific domain.
The paper tackles the problem of protecting user-level privacy in Reinforcement Learning with Human Feedback (RLHF) for large language models, proposing a novel framework called AUP-RLHF that guarantees user-level differential privacy and shows improved performance in sentiment generation and summarization tasks compared to baseline methods.
Reinforcement Learning with Human Feedback (RLHF) has emerged as an influential technique, enabling the alignment of large language models (LLMs) with human preferences. Despite the promising potential of RLHF, how to protect user preference privacy has become a crucial issue. Most previous work has focused on using differential privacy (DP) to protect the privacy of individual data. However, they have concentrated primarily on item-level privacy protection and have unsatisfactory performance for user-level privacy, which is more common in RLHF. This study proposes a novel framework, AUP-RLHF, which integrates user-level label DP into RLHF. We first show that the classical random response algorithm, which achieves an acceptable performance in item-level privacy, leads to suboptimal utility when in the user-level settings. We then establish a lower bound for the user-level label DP-RLHF and develop the AUP-RLHF algorithm, which guarantees $(\varepsilon, δ)$ user-level privacy and achieves an improved estimation error. Experimental results show that AUP-RLHF outperforms existing baseline methods in sentiment generation and summarization tasks, achieving a better privacy-utility trade-off.