Reward Dropout Improves Control: Bi-objective Perspective on Reinforced LM
This work addresses the challenge of balancing reward and likelihood in RLMs for improved control, representing an incremental advancement with a novel method for a known bottleneck.
The paper tackles the problem of optimizing Reinforced Language Models (RLMs) by framing it as a bi-objective optimization to maximize conflicting reward and likelihood objectives, and proposes Reward Dropout, which significantly improves optimization performance across five benchmark datasets and four LLMs.
We study the theoretical aspects of Reinforced Language Models (RLMs) from a bi-objective optimization perspective. Specifically, we consider the RLMs as a Pareto optimization problem that maximizes the two conflicting objectives, i.e., reward objective and likelihood objectives, simultaneously. Our main contribution consists of three parts. First, we establish the theoretical foundations of RLM as a Pareto optimization problem by presenting Reward Upper BOund (RUBO) and Pareto optimality. Our theoretical outcomes are supported by not only deductive proofs but also empirical results. Second, we propose Reward Dropout, a simple yet powerful method that guarantees to improve a bi-objective optimization of RLM. Lastly, we demonstrate that the Reward Dropout is consistently effective across five benchmark datasets and four benchmark LLMs, meaning that the Reward Dropout significantly improves the optimization performance of RLMs.