Large Language Models Still Face Challenges in Multi-Hop Reasoning with External Knowledge
This work highlights limitations in AI reasoning for tasks requiring complex knowledge integration, which is incremental as it builds on existing benchmarks and methods.
The paper investigates large language models' multi-hop reasoning abilities using external knowledge, finding that despite strong performance, they exhibit severe drawbacks and a large gap compared to humans.
We carry out a series of experiments to test large language models' multi-hop reasoning ability from three aspects: selecting and combining external knowledge, dealing with non-sequential reasoning tasks and generalising to data samples with larger numbers of hops. We test the GPT-3.5 model on four reasoning benchmarks with Chain-of-Thought prompting (and its variations). Our results reveal that despite the amazing performance achieved by large language models on various reasoning tasks, models still suffer from severe drawbacks which shows a large gap with humans.