P-Check: Advancing Personalized Reward Model via Learning to Generate Dynamic Checklist
For personalized AI systems, this addresses the limitation of static user context in reward modeling by dynamically adapting criteria to individual preferences.
P-Check introduces a personalized reward model that generates dynamic checklists for evaluation, improving reward accuracy and downstream generation, with robustness in out-of-distribution scenarios.
Recent approaches in personalized reward modeling have primarily focused on leveraging user interaction history to align model judgments with individual preferences. However, existing approaches largely treat user context as a static or implicit conditioning signal, failing to capture the dynamic and multi-faceted nature of human judgment. In this paper, we propose P-Check, a novel personalized reward modeling framework, designed to train a plug-and-play checklist generator that synthesizes dynamic evaluation criteria for guiding the reward prediction. To better align these checklists with personalized nuances, we introduce Preference-Contrastive Criterion Weighting, a training strategy that assigns saliency scores to criteria based on their discriminative power for personalized judgment. We conduct extensive experiments and demonstrate that P-Check not only improves reward accuracy but also enhances downstream personalized generation, and remains robust in OOD scenarios.