Holistic Order Prediction in Natural Scenes
This addresses the challenge of instance-wise geometry understanding in visual models for applications in computer vision, offering a more efficient solution compared to existing methods.
The paper tackles the problem of predicting occlusion and depth orderings for all instances in a scene from a single RGB image, proposing InstaFormer to achieve this in a single forward pass, reducing reliance on expensive inputs like segmentation masks and lowering inference costs from quadratic to linear.
Even in controlled settings, understanding instance-wise geometries is a challenging task for a wide range of visual models. Although specialized systems exist, modern arts rely on expensive input formats (category labels, binary segmentation masks) and inference costs (a quadratic amount of forward passes). We mitigate these limitations by proposing InstaFormer, a network capable of holistic order prediction. That is, solely given an input RGB image, InstaFormer returns the full occlusion and depth orderings for all the instances in the scene in a single forward pass. At its core, InstaFormer relies on interactions between object queries and latent mask descriptors that semantically represent the same objects while carrying complementary information. We comprehensively benchmark and ablate our approach to highlight its effectiveness. Our code and models are open-source and available at this URL: https://github.com/SNU-VGILab/InstaOrder.