Allies: Prompting Large Language Model with Beam Search
This addresses the problem of improving LLM-based pipelines for complex tasks like question answering, though it appears incremental as it builds on existing iterative prompting approaches.
The paper tackles the limitations of narrow information coverage and low fault tolerance in LLM pipeline methods by proposing ALLIES, which iteratively generates related queries to enable iterative reasoning. Experimental results on NQ, WebQ and TriviaQA benchmarks show that ALLIES significantly outperforms other zero-shot baselines in zero-shot open-domain question answering.
With the advance of large language models (LLMs), the research field of LLM applications becomes more and more popular and the idea of constructing pipelines to accomplish complex tasks by stacking LLM API calls come true. However, this kind of methods face two limitations: narrow information coverage and low fault tolerance. In this work, we propose a novel method called ALLIES. Given an input query, ALLIES leverages LLMs to iteratively generate new queries related to the original query, enabling an iterative reasoning process. By iteratively refining and expanding the scope of the original query, ALLIES captures and utilizes hidden knowledge that may not be directly obtainable through retrieval. We take zero-shot open-domain question answering (ODQA) as an application scene and evaluate ALLIES on the widely-used benchmarks, such as NQ, WebQ and TriviaQA. The experimental results demonstrate that ALLIES significantly outperforms other zero-shot baselines, indicating its effectiveness in tackling those challenges. Our code is available in https://github.com/microsoft/SimXNS/tree/main/ALLIES.