Overcoming Conflicting Data when Updating a Neural Semantic Parser
This addresses the incremental issue of efficiently updating task-oriented semantic parsers for developers when label changes occur.
The paper tackles the problem of updating a neural semantic parser with new data when conflicting out-of-date labels hinder learning, showing that multi-task and data selection methods improve accuracy by closing 86% of the gap to an oracle baseline.
In this paper, we explore how to use a small amount of new data to update a task-oriented semantic parsing model when the desired output for some examples has changed. When making updates in this way, one potential problem that arises is the presence of conflicting data, or out-of-date labels in the original training set. To evaluate the impact of this understudied problem, we propose an experimental setup for simulating changes to a neural semantic parser. We show that the presence of conflicting data greatly hinders learning of an update, then explore several methods to mitigate its effect. Our multi-task and data selection methods lead to large improvements in model accuracy compared to a naive data-mixing strategy, and our best method closes 86% of the accuracy gap between this baseline and an oracle upper bound.