DeepThink3D: Enhancing Large Language Models with Programmatic Reasoning in Complex 3D Situated Reasoning Tasks
It addresses the problem of limited reasoning capabilities in 3D scenes for AI systems, representing an incremental improvement over existing methods.
This work tackles the challenge of enhancing large language models (LLMs) for complex 3D situated reasoning by introducing DeepThink3D, which uses a combinatorial and iterative evolutionary approach to generate more complex questions on the SQA3D benchmark and fine-tunes LLMs with Direct Preference Optimization (DPO) to improve tool usage accuracy.
This work enhances the ability of large language models (LLMs) to perform complex reasoning in 3D scenes. Recent work has addressed the 3D situated reasoning task by invoking tool usage through large language models. Large language models call tools via APIs and integrate the generated programs through a chain of thought to solve problems based on the program results. However, due to the simplicity of the questions in the dataset, the generated program reasoning chains are relatively short. To solve this main challenge, in this paper, we introduce DeepThink3D to enhance the tool usage of LLMs in complex 3D situated reasoning tasks. Our work proposes a combinatorial and iterative evolutionary approach on the SQA3D benchmark to generate more complex questions. Building on this foundation, we fine-tune the large language model to make it more proficient in using 3D tools. By employing Direct Preference Optimization (DPO), we directly optimize the toolchain strategies generated by models, thereby enhancing their accuracy in complex tasks.