Factors affecting the in-context learning abilities of LLMs for dialogue state tracking
It addresses the problem of improving dialogue state tracking for conversational AI systems, but appears incremental as it focuses on analyzing existing ICL methods rather than introducing new ones.
This study investigated how in-context learning (ICL) can be applied to dialogue state tracking (DST) and analyzed factors like demonstration selection and prompt context affecting its performance, using models such as OLMo-7B-instruct on the MultiWoZ2.4 dataset.
This study explores the application of in-context learning (ICL) to the dialogue state tracking (DST) problem and investigates the factors that influence its effectiveness. We use a sentence embedding based k-nearest neighbour method to retrieve the suitable demonstrations for ICL. The selected demonstrations, along with the test samples, are structured within a template as input to the LLM. We then conduct a systematic study to analyse the impact of factors related to demonstration selection and prompt context on DST performance. This work is conducted using the MultiWoZ2.4 dataset and focuses primarily on the OLMo-7B-instruct, Mistral-7B-Instruct-v0.3, and Llama3.2-3B-Instruct models. Our findings provide several useful insights on in-context learning abilities of LLMs for dialogue state tracking.