VAM: Verbalized Action Masking for Controllable Exploration in RL Post-Training -- A Chess Case Study
This addresses the problem of sparse feedback and large action spaces in RL post-training for LLMs, though it is incremental as it builds on existing masking techniques.
The paper tackles the exploration bottleneck in reinforcement learning post-training of large language models by proposing Verbalized Action Masking (VAM), which improves learning efficiency and final performance in chess tasks, as measured by average centipawn loss.
Exploration remains a key bottleneck for reinforcement learning (RL) post-training of large language models (LLMs), where sparse feedback and large action spaces can lead to premature collapse into repetitive behaviors. We propose Verbalized Action Masking (VAM), which verbalizes an action mask in the prompt and enforces that the model outputs an action from the masked set. Building on this interface, we introduce iterative action-space pruning: if the target action is not sampled, we remove valid sampled actions from the mask and resample under the reduced candidate set, repeating until the target is sampled or a fixed budget is exhausted. We study VAM in chess and evaluate it under two training regimes: an engine-play regime that generates states via play against an engine opponent and a fixed-dataset regime that trains from a fixed dataset of positions with verifier scores. Across held-out chess puzzles and full-game play measured by average centipawn loss (ACPL), VAM improves learning efficiency and final performance over strong baselines, highlighting verbalized masking as a practical mechanism for controllable exploration in LLM RL post-training.