Countering Language Drift with Seeded Iterated Learning
This addresses a key issue in training dialogue agents for AI systems, though it appears incremental as it builds on existing finetuning pipelines.
The paper tackles the problem of language drift in goal-oriented dialogue agents, where agents lose linguistic properties while optimizing for task completion, and proposes Seeded Iterated Learning (SIL) to counter this drift and improve task performance, with evaluations showing effectiveness in both toy and natural language settings.
Pretraining on human corpus and then finetuning in a simulator has become a standard pipeline for training a goal-oriented dialogue agent. Nevertheless, as soon as the agents are finetuned to maximize task completion, they suffer from the so-called language drift phenomenon: they slowly lose syntactic and semantic properties of language as they only focus on solving the task. In this paper, we propose a generic approach to counter language drift called Seeded iterated learning (SIL). We periodically refine a pretrained student agent by imitating data sampled from a newly generated teacher agent. At each time step, the teacher is created by copying the student agent, before being finetuned to maximize task completion. SIL does not require external syntactic constraint nor semantic knowledge, making it a valuable task-agnostic finetuning protocol. We evaluate SIL in a toy-setting Lewis Game, and then scale it up to the translation game with natural language. In both settings, SIL helps counter language drift as well as it improves the task completion compared to baselines.