Neural Neural Textures Make Sim2Real Consistent
This work addresses the challenge of temporal inconsistency in sim2real applications, which is crucial for downstream tasks like robotic manipulation, representing an incremental improvement over existing unpaired image translation methods.
The paper tackles the problem of generating temporally consistent images in sim2real tasks by introducing TRITON, an unsupervised algorithm that combines differentiable rendering with image translation using neural neural textures, achieving consistency over indefinite timescales and handling both camera and object movements.
Unpaired image translation algorithms can be used for sim2real tasks, but many fail to generate temporally consistent results. We present a new approach that combines differentiable rendering with image translation to achieve temporal consistency over indefinite timescales, using surface consistency losses and \emph{neural neural textures}. We call this algorithm TRITON (Texture Recovering Image Translation Network): an unsupervised, end-to-end, stateless sim2real algorithm that leverages the underlying 3D geometry of input scenes by generating realistic-looking learnable neural textures. By settling on a particular texture for the objects in a scene, we ensure consistency between frames statelessly. Unlike previous algorithms, TRITON is not limited to camera movements -- it can handle the movement of objects as well, making it useful for downstream tasks such as robotic manipulation.