Few-Shot Adaptation for Parsing Contextual Utterances with LLMs
This addresses the challenge of few-shot adaptation for conversational semantic parsing, which is incremental as it compares existing paradigms on a new dataset.
The paper tackled the problem of adapting semantic parsers based on large language models to handle contextual utterances with limited annotated data, finding that the Rewrite-then-Parse paradigm performed best in terms of parsing accuracy, annotation cost, and error types.
We evaluate the ability of semantic parsers based on large language models (LLMs) to handle contextual utterances. In real-world settings, there typically exists only a limited number of annotated contextual utterances due to annotation cost, resulting in an imbalance compared to non-contextual utterances. Therefore, parsers must adapt to contextual utterances with a few training examples. We examine four major paradigms for doing so in conversational semantic parsing i.e., Parse-with-Utterance-History, Parse-with-Reference-Program, Parse-then-Resolve, and Rewrite-then-Parse. To facilitate such cross-paradigm comparisons, we construct SMCalFlow-EventQueries, a subset of contextual examples from SMCalFlow with additional annotations. Experiments with in-context learning and fine-tuning suggest that Rewrite-then-Parse is the most promising paradigm when holistically considering parsing accuracy, annotation cost, and error types.