IB-RL: Isolated Bilateral Reinforcement Learning for Strategic Dialogue Agents
This work addresses a key limitation in applying RL to strategic dialogue, offering a method that improves generalization to unseen counterparts, which is crucial for real-world conversational AI.
The paper identifies the static-counterpart mismatch in RL for strategic dialogue, where training against a fixed counterpart leads to overfitting and poor generalization. They propose Isolated Bilateral Reinforcement Learning (IB-RL), which coevolves both roles with independent updates, achieving 89.6% Success@1 on Vehicle TeleSales (vs 84.6% baseline) and 98.4% agreement on Deal-or-NoDeal (vs 86.4% baseline).
Reinforcement learning (RL) has achieved strong results in improving large language models (LLMs) on tasks with stationary, verifiable rewards, such as mathematical reasoning and code execution. In these settings, the environment follows fixed rules and does not adapt strategically to the agent. Strategic dialogue differs in this respect: the environment is another agent that adapts to the policy, and success depends on the interaction between the two sides. Despite this interactive nature, current RL approaches typically train a target agent against a fixed counterpart or simulator. We find that this training paradigm encourages the policy to exploit counterpart-specific regularities rather than learn strategies that generalize across counterparts. We call this problem the static-counterpart mismatch, which we quantify directly in our experiments. To address it, we propose Isolated Bilateral Reinforcement Learning (IB-RL), in which the two roles coevolve through joint rollouts while each role optimizes its own reward through fully independent advantages, action masks, and update paths. We evaluate frozen policies against fully independent held-out counterparts in both domains. On Vehicle TeleSales, IB-RL achieves 89.6% Success@1, compared to 84.6% for the best unilateral RL baseline. On Deal-or-NoDeal, it reaches 98.4% agreement against DeepSeek V4 Pro, compared to 86.4% for the best unilateral baseline. These results indicate that jointly training both roles with strict peragent isolation produces policies that generalize more effectively to unseen counterparts.