Are LLMs good pragmatic speakers?
This work addresses the problem of understanding LLM capabilities in pragmatic reasoning for researchers in computational linguistics and AI, but it is incremental as it builds on existing frameworks and calls for further validation.
The study investigated whether large language models (LLMs) behave like pragmatic speakers by comparing them to the Rational Speech Act (RSA) framework in a reference game using the TUNA corpus, finding only some positive correlation and insufficient evidence for pragmatic behavior.
Large language models (LLMs) are trained on data assumed to include natural language pragmatics, but do they actually behave like pragmatic speakers? We attempt to answer this question using the Rational Speech Act (RSA) framework, which models pragmatic reasoning in human communication. Using the paradigm of a reference game constructed from the TUNA corpus, we score candidate referential utterances in both a state-of-the-art LLM (Llama3-8B-Instruct) and in the RSA model, comparing and contrasting these scores. Given that RSA requires defining alternative utterances and a truth-conditional meaning function, we explore such comparison for different choices of each of these requirements. We find that while scores from the LLM have some positive correlation with those from RSA, there isn't sufficient evidence to claim that it behaves like a pragmatic speaker. This initial study paves way for further targeted efforts exploring different models and settings, including human-subject evaluation, to see if LLMs truly can, or be made to, behave like pragmatic speakers.