MemReward: Graph-Based Experience Memory for LLM Reward Prediction with Limited Labels
This addresses the high cost of human labeling for LLM fine-tuning in tasks like mathematics and question answering, offering a scalable solution with incremental improvements over existing methods.
The paper tackles the problem of limited reward labels for training large language models via reinforcement learning by introducing MemReward, a graph-based experience memory framework that propagates rewards to unlabeled rollouts using a GNN, achieving 97.3% of Oracle performance with only 20% labels on a 3B model and scaling to 99.4% at 70% labels.
Training large language models (LLMs) for complex reasoning via reinforcement learning requires reward labels that specify whether the generated rollouts are correct. However, obtaining reward labels at scale often requires expensive human labeling or time-consuming verification procedures; for instance, evaluating mathematical proofs demands expert review, while open-ended question answering lacks definitive ground truth. When reward labels are limited, the effectiveness of reinforcement learning fine-tuning is constrained by the scarcity of reward labels. We introduce MemReward, a graph-based experience memory framework: an initial LLM policy generates rollouts for each query, each comprising a thinking process and a final answer, and these rollouts are stored as experience memory. Queries, thinking processes, and answers form nodes in a heterogeneous graph with similarity and structural edges; a GNN trained on labeled nodes propagates rewards to unlabeled rollouts during online optimization. Experiments on Qwen2.5-3B and 1.5B across mathematics, question answering, and code generation demonstrate that MemReward, with only 20% labels, achieves 97.3% of Oracle performance on 3B and 96.6% on 1.5B, surpassing Oracle on out-of-domain tasks. Performance scales smoothly with label budget, reaching 99.4% of Oracle at 70% labels.