3DAxiesPrompts: Unleashing the 3D Spatial Task Capabilities of GPT-4V
This work addresses the challenge of enhancing GPT-4V's capabilities for 3D spatial reasoning, which is incremental as it builds on existing visual prompting techniques to handle a new type of task.
The authors tackled the problem of GPT-4V's unexplored abilities in 3D spatial tasks by introducing 3DAxiesPrompts (3DAP), a visual prompting method that creates a 3D coordinate system with annotated scales, enabling GPT-4V to perform tasks like 2D to 3D point reconstruction, matching, and 3D object detection with high precision on their proposed dataset 3DAP-Data.
In this work, we present a new visual prompting method called 3DAxiesPrompts (3DAP) to unleash the capabilities of GPT-4V in performing 3D spatial tasks. Our investigation reveals that while GPT-4V exhibits proficiency in discerning the position and interrelations of 2D entities through current visual prompting techniques, its abilities in handling 3D spatial tasks have yet to be explored. In our approach, we create a 3D coordinate system tailored to 3D imagery, complete with annotated scale information. By presenting images infused with the 3DAP visual prompt as inputs, we empower GPT-4V to ascertain the spatial positioning information of the given 3D target image with a high degree of precision. Through experiments, We identified three tasks that could be stably completed using the 3DAP method, namely, 2D to 3D Point Reconstruction, 2D to 3D point matching, and 3D Object Detection. We perform experiments on our proposed dataset 3DAP-Data, the results from these experiments validate the efficacy of 3DAP-enhanced GPT-4V inputs, marking a significant stride in 3D spatial task execution.