A LLM Benchmark based on the Minecraft Builder Dialog Agent Task
This provides a domain-specific benchmark for researchers and developers working on LLM-based builder agents in spatial tasks, though it is incremental as it builds on existing Minecraft task adaptations.
The authors tackled the problem of evaluating LLMs in spatial reasoning by adapting the Minecraft builder task into a synthetic benchmark, resulting in a comprehensive test suite for builder agents that probes specific strengths and weaknesses.
In this work we proposing adapting the Minecraft builder task into an LLM benchmark suitable for evaluating LLM ability in spatially orientated tasks, and informing builder agent design. Previous works have proposed corpora with varying complex structures, and human written instructions. We instead attempt to provide a comprehensive synthetic benchmark for testing builder agents over a series of distinct tasks that comprise of common building operations. We believe this approach allows us to probe specific strengths and weaknesses of different agents, and test the ability of LLMs in the challenging area of spatial reasoning and vector based math.