The Change You Want to See (Now in 3D)
This work addresses the problem of open-set change detection in 3D scenes for applications like surveillance or monitoring, but it is incremental as it builds on existing 'register and difference' approaches with new features.
The paper tackles the problem of detecting changes between two images of the same 3D scene taken from different viewpoints and times, addressing challenges like occlusions and lack of training data, and achieves a model trained on synthetic data that performs well on real-world images without fine-tuning.
The goal of this paper is to detect what has changed, if anything, between two "in the wild" images of the same 3D scene acquired from different camera positions and at different temporal instances. The open-set nature of this problem, occlusions/dis-occlusions due to the shift in viewpoint, and the lack of suitable training datasets, presents substantial challenges in devising a solution. To address this problem, we contribute a change detection model that is trained entirely on synthetic data and is class-agnostic, yet it is performant out-of-the-box on real world images without requiring fine-tuning. Our solution entails a "register and difference" approach that leverages self-supervised frozen embeddings and feature differences, which allows the model to generalise to a wide variety of scenes and domains. The model is able to operate directly on two RGB images, without requiring access to ground truth camera intrinsics, extrinsics, depth maps, point clouds, or additional before-after images. Finally, we collect and release a new evaluation dataset consisting of real-world image pairs with human-annotated differences and demonstrate the efficacy of our method. The code, datasets and pre-trained model can be found at: https://github.com/ragavsachdeva/CYWS-3D