Distracted Robot: How Visual Clutter Undermine Robotic Manipulation
This addresses the challenge of robust robotic manipulation in real-world environments for robotics researchers, though it is incremental as it focuses on evaluation rather than new policy development.
The paper tackles the problem of evaluating robotic manipulation policies in cluttered scenes by proposing a psychophysical evaluation protocol with a unified clutter measure, finding that scene clutter can lower policy performance by up to 34% and that different vision-language-action models have unique vulnerabilities.
In this work, we propose an evaluation protocol for examining the performance of robotic manipulation policies in cluttered scenes. Contrary to prior works, we approach evaluation from a psychophysical perspective, therefore we use a unified measure of clutter that accounts for environmental factors as well as the distractors quantity, characteristics, and arrangement. Using this measure, we systematically construct evaluation scenarios in both hyper-realistic simulation and real-world and conduct extensive experimentation on manipulation policies, in particular vision-language-action (VLA) models. Our experiments highlight the significant impact of scene clutter, lowering the performance of the policies, by as much as 34% and show that despite achieving similar average performance across the tasks, different VLA policies have unique vulnerabilities and a relatively low agreement on success scenarios. We further show that our clutter measure is an effective indicator of performance degradation and analyze the impact of distractors in terms of their quantity and occluding influence. At the end, we show that finetuning on enhanced data, although effective, does not equally remedy all negative impacts of clutter on performance.