Multi-Object Manipulation via Object-Centric Neural Scattering Functions
This work addresses the challenge of precise modeling and manipulation in multi-object environments with varying lighting for robotics, representing an incremental advancement over existing object decomposition methods.
The paper tackled the problem of representing scenes for multi-object robotic manipulation under varying lighting by proposing object-centric neural scattering functions (OSFs) to model per-object light transport, resulting in improved model-predictive control performance and generalization in unseen scenarios and harsh lighting conditions.
Learned visual dynamics models have proven effective for robotic manipulation tasks. Yet, it remains unclear how best to represent scenes involving multi-object interactions. Current methods decompose a scene into discrete objects, but they struggle with precise modeling and manipulation amid challenging lighting conditions as they only encode appearance tied with specific illuminations. In this work, we propose using object-centric neural scattering functions (OSFs) as object representations in a model-predictive control framework. OSFs model per-object light transport, enabling compositional scene re-rendering under object rearrangement and varying lighting conditions. By combining this approach with inverse parameter estimation and graph-based neural dynamics models, we demonstrate improved model-predictive control performance and generalization in compositional multi-object environments, even in previously unseen scenarios and harsh lighting conditions.