Neural Textured Deformable Meshes for Robust Analysis-by-Synthesis
This addresses robustness issues in AI vision systems for applications requiring reliable performance in varied environments, though it appears incremental as it builds on existing analysis-by-synthesis concepts.
The paper tackles robustness in vision tasks under out-of-distribution scenarios by introducing Neural Textured Deformable Meshes for analysis-by-synthesis, showing improved robustness compared to conventional neural networks in real-world images and challenging conditions like occlusion and domain shift.
Human vision demonstrates higher robustness than current AI algorithms under out-of-distribution scenarios. It has been conjectured such robustness benefits from performing analysis-by-synthesis. Our paper formulates triple vision tasks in a consistent manner using approximate analysis-by-synthesis by render-and-compare algorithms on neural features. In this work, we introduce Neural Textured Deformable Meshes, which involve the object model with deformable geometry that allows optimization on both camera parameters and object geometries. The deformable mesh is parameterized as a neural field, and covered by whole-surface neural texture maps, which are trained to have spatial discriminability. During inference, we extract the feature map of the test image and subsequently optimize the 3D pose and shape parameters of our model using differentiable rendering to best reconstruct the target feature map. We show that our analysis-by-synthesis is much more robust than conventional neural networks when evaluated on real-world images and even in challenging out-of-distribution scenarios, such as occlusion and domain shift. Our algorithms are competitive with standard algorithms when tested on conventional performance measures.