PersRM-R1: Enhance Personalized Reward Modeling with Reinforcement Learning
This addresses the challenge of personalizing LLMs for users with limited data, though it is an incremental improvement on existing reward modeling techniques.
The paper tackles the problem of reward models failing to capture nuanced user-specific preferences with limited data by introducing PersRM-R1, a reasoning-based reward modeling framework that uses synthetic data generation and a two-stage training pipeline; it outperforms similar-sized models and matches larger models in accuracy and generalizability.
Reward models (RMs), which are central to existing post-training methods, aim to align LLM outputs with human values by providing feedback signals during fine-tuning. However, existing RMs struggle to capture nuanced, user-specific preferences, especially under limited data and across diverse domains. Thus, we introduce PersRM-R1, the first reasoning-based reward modeling framework specifically designed to identify and represent personal factors from only one or a few personal exemplars. To address challenges including limited data availability and the requirement for robust generalization, our approach combines synthetic data generation with a two-stage training pipeline consisting of supervised fine-tuning followed by reinforcement fine-tuning. Experimental results demonstrate that PersRM-R1 outperforms existing models of similar size and matches the performance of much larger models in both accuracy and generalizability, paving the way for more effective personalized LLMs.