DiffVL: Scaling Up Soft Body Manipulation using Vision-Language Driven Differentiable Physics
This addresses the difficulty for non-experts in specifying complex soft-body manipulation tasks, making the technology more accessible, though it is incremental as it builds on existing differentiable physics methods.
The paper tackles the problem of requiring expert knowledge to write optimization objectives for soft-body manipulation tasks in differentiable physics, introducing DiffVL which uses vision-language input from non-experts to generate objectives, enabling the solver to handle 100 real-life inspired tasks that were challenging for previous baselines.
Combining gradient-based trajectory optimization with differentiable physics simulation is an efficient technique for solving soft-body manipulation problems. Using a well-crafted optimization objective, the solver can quickly converge onto a valid trajectory. However, writing the appropriate objective functions requires expert knowledge, making it difficult to collect a large set of naturalistic problems from non-expert users. We introduce DiffVL, a method that enables non-expert users to communicate soft-body manipulation tasks -- a combination of vision and natural language, given in multiple stages -- that can be readily leveraged by a differential physics solver. We have developed GUI tools that enable non-expert users to specify 100 tasks inspired by real-life soft-body manipulations from online videos, which we'll make public. We leverage large language models to translate task descriptions into machine-interpretable optimization objectives. The optimization objectives can help differentiable physics solvers to solve these long-horizon multistage tasks that are challenging for previous baselines.