Exploring State Tracking Capabilities of Large Language Models
This work addresses the state tracking capability for LLM users, but it is incremental as it builds on existing benchmarks and methods.
The paper tackled the problem of state tracking in large language models (LLMs) by proposing a benchmark based on three well-defined tasks, finding that recent LLMs like GPT-4 and Llama3 can track state effectively, especially with Chain of Thought, while older models often fail after multiple steps.
Large Language Models (LLMs) have demonstrated impressive capabilities in solving complex tasks, including those requiring a certain level of reasoning. In this paper, we focus on state tracking, a problem where models need to keep track of the state governing a number of entities. To isolate the state tracking component from other factors, we propose a benchmark based on three well-defined state tracking tasks and analyse the performance of LLMs in different scenarios. The results indicate that the recent generation of LLMs (specifically, GPT-4 and Llama3) are capable of tracking state, especially when integrated with mechanisms such as Chain of Thought. However, models from the former generation, while understanding the task and being able to solve it at the initial stages, often fail at this task after a certain number of steps.