RESPONSE: Benchmarking the Ability of Language Models to Undertake Commonsense Reasoning in Crisis Situation
This addresses the need for better AI reasoning in crisis situations, though it is incremental as it focuses on benchmarking rather than solving the reasoning problem directly.
The authors tackled the problem of evaluating language models' commonsense reasoning in disaster scenarios by creating the RESPONSE dataset with 1,789 annotated instances, finding that even GPT-4 achieved only 37% human-evaluated correctness for immediate response actions.
An interesting class of commonsense reasoning problems arises when people are faced with natural disasters. To investigate this topic, we present \textsf{RESPONSE}, a human-curated dataset containing 1789 annotated instances featuring 6037 sets of questions designed to assess LLMs' commonsense reasoning in disaster situations across different time frames. The dataset includes problem descriptions, missing resources, time-sensitive solutions, and their justifications, with a subset validated by environmental engineers. Through both automatic metrics and human evaluation, we compare LLM-generated recommendations against human responses. Our findings show that even state-of-the-art models like GPT-4 achieve only 37\% human-evaluated correctness for immediate response actions, highlighting significant room for improvement in LLMs' ability for commonsense reasoning in crises.