Integrating Large Language Models and Reinforcement Learning for Non-Linear Reasoning
This addresses a specific bottleneck in LLMs for tasks requiring non-linear reasoning, but it is incremental as it builds on existing methods like CoT and ToT.
The paper tackled the problem of large language models struggling with long-term planning by integrating reinforcement learning to guide exploration, resulting in positive comparisons against Chain of Thought and Tree of Thoughts on a program equivalence task.
Large Language Models (LLMs) were shown to struggle with long-term planning, which may be caused by the limited way in which they explore the space of possible solutions. We propose an architecture where a Reinforcement Learning (RL) Agent guides an LLM's space exploration: (1) the Agent has access to domain-specific information, and can therefore make decisions about the quality of candidate solutions based on specific and relevant metrics, which were not explicitly considered by the LLM's training objective; (2) the LLM can focus on generating immediate next steps, without the need for long-term planning. We allow non-linear reasoning by exploring alternative paths and backtracking. We evaluate this architecture on the program equivalence task, and compare it against Chain of Thought (CoT) and Tree of Thoughts (ToT). We assess both the downstream task, denoting the binary classification, and the intermediate reasoning steps. Our approach compares positively against CoT and ToT.