Just Say What You Want: Only-prompting Self-rewarding Online Preference Optimization
This addresses the problem of expensive feedback in online RLHF for AI alignment, though it appears incremental as it builds on existing self-rewarding methods.
The paper tackles the challenge of costly online RLHF by proposing an only-prompting self-rewarding algorithm that generates preference datasets without relying on judgment capabilities, achieving 34.5% in Length-controlled Win Rates on AlpacaEval 2.0 with Mistral-7B models.
We address the challenge of online Reinforcement Learning from Human Feedback (RLHF) with a focus on self-rewarding alignment methods. In online RLHF, obtaining feedback requires interaction with the environment, which can be costly when using additional reward models or the GPT-4 API. Current self-rewarding approaches rely heavily on the discriminator's judgment capabilities, which are effective for large-scale models but challenging to transfer to smaller ones. To address these limitations, we propose a novel, only-prompting self-rewarding online algorithm that generates preference datasets without relying on judgment capabilities. Additionally, we employ fine-grained arithmetic control over the optimality gap between positive and negative examples, generating more hard negatives in the later stages of training to help the model better capture subtle human preferences. Finally, we conduct extensive experiments on two base models, Mistral-7B and Mistral-Instruct-7B, which significantly bootstrap the performance of the reference model, achieving 34.5% in the Length-controlled Win Rates of AlpacaEval 2.0.