$\mathbf{(N,K)}$-Puzzle: A Cost-Efficient Testbed for Benchmarking Reinforcement Learning Algorithms in Generative Language Model
This provides a benchmark for researchers working on RL algorithms for language models, though it is incremental as it adapts an existing puzzle concept.
The paper tackles the lack of a cost-effective and standardized testbed for evaluating reinforcement learning algorithms in generative language models by introducing the $(N,K)$-Puzzle, a generalized version of the 24-Puzzle, and demonstrates its effectiveness by testing algorithms like PPO, IPO, and DPO.
Recent advances in reinforcement learning (RL) algorithms aim to enhance the performance of language models at scale. Yet, there is a noticeable absence of a cost-effective and standardized testbed tailored to evaluating and comparing these algorithms. To bridge this gap, we present a generalized version of the 24-Puzzle: the $(N,K)$-Puzzle, which challenges language models to reach a target value $K$ with $N$ integers. We evaluate the effectiveness of established RL algorithms such as Proximal Policy Optimization (PPO), alongside novel approaches like Identity Policy Optimization (IPO) and Direct Policy Optimization (DPO).