Beyond the Patch: Exploring Vulnerabilities of Visuomotor Policies via Viewpoint-Consistent 3D Adversarial Object
This work addresses a critical security and safety problem for robots controlled by visuomotor policies, specifically their susceptibility to 3D adversarial attacks, which is an incremental step beyond 2D patches.
This paper explores vulnerabilities of visuomotor policies to 3D adversarial objects, proposing a viewpoint-consistent adversarial texture optimization method. The method uses differentiable rendering, Expectation over Transformation (EOT) with a Coarse-to-Fine (C2F) curriculum, and saliency-guided perturbations to create textures that consistently drive robots toward adversarial objects across varying camera-object distances and environmental conditions.
Neural network-based visuomotor policies enable robots to perform manipulation tasks but remain susceptible to perceptual attacks. For example, conventional 2D adversarial patches are effective under fixed-camera setups, where appearance is relatively consistent; however, their efficacy often diminishes under dynamic viewpoints from moving cameras, such as wrist-mounted setups, due to perspective distortions. To proactively investigate potential vulnerabilities beyond 2D patches, this work proposes a viewpoint-consistent adversarial texture optimization method for 3D objects through differentiable rendering. As optimization strategies, we employ Expectation over Transformation (EOT) with a Coarse-to-Fine (C2F) curriculum, exploiting distance-dependent frequency characteristics to induce textures effective across varying camera-object distances. We further integrate saliency-guided perturbations to redirect policy attention and design a targeted loss that persistently drives robots toward adversarial objects. Our comprehensive experiments show that the proposed method is effective under various environmental conditions, while confirming its black-box transferability and real-world applicability.