Which objects help me to act effectively? Reasoning about physically-grounded affordances
This work addresses the challenge of open-world interaction for robots by improving affordance detection, though it is incremental as it builds on existing LLM and VLM methods.
The paper tackled the problem of enabling robots to identify useful objects for achieving goals by detecting object affordances, and demonstrated that grounding in physical properties and robot embodiment improved performance, including selecting useful objects from distractors.
For effective interactions with the open world, robots should understand how interactions with known and novel objects help them towards their goal. A key aspect of this understanding lies in detecting an object's affordances, which represent the potential effects that can be achieved by manipulating the object in various ways. Our approach leverages a dialogue of large language models (LLMs) and vision-language models (VLMs) to achieve open-world affordance detection. Given open-vocabulary descriptions of intended actions and effects, the useful objects in the environment are found. By grounding our system in the physical world, we account for the robot's embodiment and the intrinsic properties of the objects it encounters. In our experiments, we have shown that our method produces tailored outputs based on different embodiments or intended effects. The method was able to select a useful object from a set of distractors. Finetuning the VLM for physical properties improved overall performance. These results underline the importance of grounding the affordance search in the physical world, by taking into account robot embodiment and the physical properties of objects.