Reasoning about Ambiguous Definite Descriptions
This work addresses the problem of evaluating reasoning in LLMs for researchers and developers, but it is incremental as it introduces a new dataset without proposing a novel method.
The authors tackled the lack of resources for evaluating how well Large Language Models (LLMs) use explicit reasoning to resolve ambiguity in language by creating the first benchmark dataset of ambiguous definite descriptions, finding it to be a challenging task for recent LLMs.
Natural language reasoning plays an increasingly important role in improving language models' ability to solve complex language understanding tasks. An interesting use case for reasoning is the resolution of context-dependent ambiguity. But no resources exist to evaluate how well Large Language Models can use explicit reasoning to resolve ambiguity in language. We propose to use ambiguous definite descriptions for this purpose and create and publish the first benchmark dataset consisting of such phrases. Our method includes all information required to resolve the ambiguity in the prompt, which means a model does not require anything but reasoning to do well. We find this to be a challenging task for recent LLMs. Code and data available at: https://github.com/sfschouten/exploiting-ambiguity