Detecting Referring Expressions in Visually Grounded Dialogue with Autoregressive Language Models
This work addresses the challenge of extracting referring expressions from dialogue for multimodal AI systems, but it is incremental as it adapts existing methods to a specific task.
The paper tackled the problem of detecting referring expressions in visually grounded dialogue using a text-only autoregressive language model, finding that this approach can be effective with small datasets and parameter-efficient fine-tuning, though it acknowledges inherent multimodal limitations.
In this paper, we explore the use of a text-only, autoregressive language modeling approach for the extraction of referring expressions from visually grounded dialogue. More specifically, the aim is to investigate the extent to which the linguistic context alone can inform the detection of mentions that have a (visually perceivable) referent in the visual context of the conversation. To this end, we adapt a pretrained large language model (LLM) to perform a relatively course-grained annotation of mention spans in unfolding conversations by demarcating mention span boundaries in text via next-token prediction. Our findings indicate that even when using a moderately sized LLM, relatively small datasets, and parameter-efficient fine-tuning, a text-only approach can be effective, highlighting the relative importance of the linguistic context for this task. Nevertheless, we argue that the task represents an inherently multimodal problem and discuss limitations fundamental to unimodal approaches.