Verbalized Algorithms
This addresses the challenge of improving reliability in LLM-based reasoning for AI practitioners, though it is incremental as it builds on classical algorithms.
The paper tackles the problem of unreliable LLM reasoning by proposing verbalized algorithms, which decompose tasks into simple operations like binary comparisons, and demonstrates effectiveness on sorting and clustering tasks.
Instead of querying LLMs in a one-shot manner and hoping to get the right answer for a reasoning task, we propose a paradigm we call \emph{verbalized algorithms} (VAs), which leverage classical algorithms with established theoretical understanding. VAs decompose a task into simple elementary operations on natural language strings that they should be able to answer reliably, and limit the scope of LLMs to only those simple tasks. For example, for sorting a series of natural language strings, \emph{verbalized sorting} uses an LLM as a binary comparison oracle in a known and well-analyzed sorting algorithm (e.g., bitonic sorting network). We demonstrate the effectiveness of this approach on sorting and clustering tasks.