RPNR: Robust-Perception Neural Reshading
This addresses the challenge of realistic object insertion in AR for scenarios with limited object information, though it appears incremental as it builds on existing neural rendering techniques.
The paper tackles the problem of inserting unknown objects from a source image into a target scene for AR applications without requiring predefined 3D models or labeled data, achieving coherence through a method that uses only two images and compares qualitatively to baselines.
Augmented Reality (AR) applications necessitates methods of inserting needed objects into scenes captured by cameras in a way that is coherent with the surroundings. Common AR applications require the insertion of predefined 3D objects with known properties and shape. This simplifies the problem since it is reduced to extracting an illumination model for the object in that scene by understanding the surrounding light sources. However, it is often not the case that we have information about the properties of an object, especially when we depart from a single source image. Our method renders such source fragments in a coherent way with the target surroundings using only these two images. Our pipeline uses a Deep Image Prior (DIP) network based on a U-Net architecture as the main renderer, alongside robust-feature extracting networks that are used to apply needed losses. Our method does not require any pair-labeled data, and no extensive training on a dataset. We compare our method using qualitative metrics to the baseline methods such as Cut and Paste, Cut And Paste Neural Rendering, and Image Harmonization