Parameter Efficient Reinforcement Learning from Human Feedback
This work addresses the computational burden of RLHF for aligning large language and vision-language models, making it more accessible, though it is incremental as it builds on existing parameter-efficient methods like LoRA.
The paper tackles the high computational cost of Reinforcement Learning from Human Feedback (RLHF) by introducing Parameter Efficient RLHF (PE-RLHF) using LoRA fine-tuning, achieving comparable performance while reducing training time by up to 90% for reward models and 30% for RL, and memory footprint by up to 50% for reward models and 27% for RL.
While Reinforcement Learning from Human Feedback (RLHF) effectively aligns pretrained Large Language and Vision-Language Models (LLMs, and VLMs) with human preferences, its computational cost and complexity hamper its wider adoption. To alleviate some of the computational burden of fine-tuning, parameter efficient methods, like LoRA were introduced. In this work, we empirically evaluate the setup of Parameter Efficient Reinforcement Learning from Human Feedback (PE-RLHF) that leverages LoRA fine-tuning for Reward Modeling, and Reinforcement Learning. We benchmark the PE-RLHF setup on six diverse datasets spanning summarization, harmless/helpful response generation, UI automation, and visual question answering in terms of effectiveness of the trained models, and the training resources required. Our findings show, for the first time, that PE-RLHF achieves comparable performance to RLHF, while significantly reducing training time (up to 90% faster for reward models, and 30% faster for RL), and memory footprint (up to 50% reduction for reward models, and 27% for RL). We provide comprehensive ablations across LoRA ranks, and model sizes for both reward modeling and reinforcement learning. By mitigating the computational burden associated with RLHF, we push for a broader adoption of PE-RLHF as an alignment technique for LLMs and VLMs.