ReMOVE: A Reference-free Metric for Object Erasure
This addresses a key challenge in practical image inpainting scenarios for AI researchers and developers, though it is incremental as it builds on existing metrics.
The paper tackles the problem of evaluating object erasure in diffusion-based image editing models without a reference image, introducing ReMOVE as a novel metric that effectively distinguishes between removal and replacement and aligns with human perception.
We introduce $\texttt{ReMOVE}$, a novel reference-free metric for assessing object erasure efficacy in diffusion-based image editing models post-generation. Unlike existing measures such as LPIPS and CLIPScore, $\texttt{ReMOVE}$ addresses the challenge of evaluating inpainting without a reference image, common in practical scenarios. It effectively distinguishes between object removal and replacement. This is a key issue in diffusion models due to stochastic nature of image generation. Traditional metrics fail to align with the intuitive definition of inpainting, which aims for (1) seamless object removal within masked regions (2) while preserving the background continuity. $\texttt{ReMOVE}$ not only correlates with state-of-the-art metrics and aligns with human perception but also captures the nuanced aspects of the inpainting process, providing a finer-grained evaluation of the generated outputs.