Integrating Neural and Symbolic Components in a Model of Pragmatic Question-Answering
This addresses the scalability problem for researchers building computational models of pragmatic language use, representing an incremental improvement through hybrid integration.
The authors tackled the problem of computational models of pragmatic language use being limited by hand-specified utterances and meanings, proposing a neuro-symbolic framework that integrates LLM-based modules to eliminate manual specification. They found that hybrid models can match or exceed traditional probabilistic models in predicting human answer patterns in pragmatic question-answering, though success depends on how LLMs are integrated.
Computational models of pragmatic language use have traditionally relied on hand-specified sets of utterances and meanings, limiting their applicability to real-world language use. We propose a neuro-symbolic framework that enhances probabilistic cognitive models by integrating LLM-based modules to propose and evaluate key components in natural language, eliminating the need for manual specification. Through a classic case study of pragmatic question-answering, we systematically examine various approaches to incorporating neural modules into the cognitive model -- from evaluating utilities and literal semantics to generating alternative utterances and goals. We find that hybrid models can match or exceed the performance of traditional probabilistic models in predicting human answer patterns. However, the success of the neuro-symbolic model depends critically on how LLMs are integrated: while they are particularly effective for proposing alternatives and transforming abstract goals into utilities, they face challenges with truth-conditional semantic evaluation. This work charts a path toward more flexible and scalable models of pragmatic language use while illuminating crucial design considerations for balancing neural and symbolic components.