A Knowledge Hunting Framework for Common Sense Reasoning
This addresses the problem of common sense reasoning for AI systems, representing a strong specific gain in performance.
The paper tackles the Winograd Schema Challenge, a common sense reasoning task, by introducing an automatic system that uses a knowledge hunting module to gather web text as evidence, achieving state-of-the-art results with a 0.21 F1 improvement and exceeding 0.5 F1 for the first time.
We introduce an automatic system that achieves state-of-the-art results on the Winograd Schema Challenge (WSC), a common sense reasoning task that requires diverse, complex forms of inference and knowledge. Our method uses a knowledge hunting module to gather text from the web, which serves as evidence for candidate problem resolutions. Given an input problem, our system generates relevant queries to send to a search engine, then extracts and classifies knowledge from the returned results and weighs them to make a resolution. Our approach improves F1 performance on the full WSC by 0.21 over the previous best and represents the first system to exceed 0.5 F1. We further demonstrate that the approach is competitive on the Choice of Plausible Alternatives (COPA) task, which suggests that it is generally applicable.