Overinformative Question Answering by Humans and Machines
This addresses the problem of making AI question-answering more human-like and context-aware, but it is incremental as it builds on existing benchmarks and models.
The paper investigates the principles guiding overinformative answers to polar questions, finding that humans adjust based on relevance to the questioner's goals, and evaluates neural language models against this benchmark, showing that most models fail to adjust human-like and GPT-3 only achieves this with specific prompting.
When faced with a polar question, speakers often provide overinformative answers going beyond a simple "yes" or "no". But what principles guide the selection of additional information? In this paper, we provide experimental evidence from two studies suggesting that overinformativeness in human answering is driven by considerations of relevance to the questioner's goals which they flexibly adjust given the functional context in which the question is uttered. We take these human results as a strong benchmark for investigating question-answering performance in state-of-the-art neural language models, conducting an extensive evaluation on items from human experiments. We find that most models fail to adjust their answering behavior in a human-like way and tend to include irrelevant information. We show that GPT-3 is highly sensitive to the form of the prompt and only achieves human-like answer patterns when guided by an example and cognitively-motivated explanation.