State Machine of Thoughts: Leveraging Past Reasoning Trajectories for Enhancing Problem Solving
This work addresses a specific inefficiency in LLM-based agents for exploration-intensive problems, offering an incremental improvement over existing methods.
The paper tackles the inefficiency of Large Language Model-based agents in reusing successful reasoning trajectories for future problems by introducing a state machine to record and leverage past trajectories, resulting in significant improvements in problem-solving abilities for the 24-point game and a taxi navigation game.
Current Large Language Model-based agents reason within an exploration-evaluation framework, navigating problem-solving processes in a tree-like manner. However, these methods often neglect successful reasoning trajectories once a problem is resolved, leading to inefficient use of these trajectories for future analogous problems. To address this inefficiency, we adopt a state machine to record experience derived from previous reasoning trajectories. Within the state machine, states represent decomposed sub-problems, while state transitions reflect the dependencies among sub-problems. The state machine records both successful and failed trajectories. Utilizing the experience from the state machine, our proposed State Machine of Thoughts (SMoT) selects the most optimal sub-solutions and avoids incorrect ones. Our experiments show that SMoT can significantly improve problem-solving abilities in two exploration-intensive problems: the 24-point game and a taxi navigation reinforcement learning game.