Parent-Guided Semantic Reward Model (PGSRM): Embedding-Based Reward Functions for Reinforcement Learning of Transformer Language Models
This provides a lightweight, annotation-free alternative to RLHF-style reward modeling for aligning smaller transformer models, though it is incremental as it builds on existing embedding and RL techniques.
The paper tackled the problem of reward modeling for reinforcement learning of transformer language models by introducing the Parent-Guided Semantic Reward Model (PGSRM), which uses cosine similarity between parent and child model embeddings as a reward signal, resulting in smoother reward improvement and more stable PPO dynamics compared to binary rewards on five language tasks.
We introduce the Parent-Guided Semantic Reward Model (PGSRM), a lightweight reward framework for reinforcement learning (RL) of transformer language models. PGSRM replaces binary correctness signals, human preference data, and trained reward models with a simple signal: cosine similarity between a parent model's reference output embedding and a child model's generated output for the same input. This yields a dense, semantically meaningful reward with no human annotation or additional model training. We apply PGSRM on five language tasks and find that it produces smoother reward improvement and more stable PPO dynamics than a binary reward baseline, suggesting that embedding-based semantic rewards are a practical alternative to RLHF-style reward modeling for parent-guided alignment in smaller transformer models.