Mapping the Mind of an Instruction-based Image Editing using SMILE
This addresses the problem of user trust in critical applications like healthcare and autonomous driving, but it is incremental as it builds on existing interpretability techniques.
The authors tackled the lack of transparency in instruction-based image editing models by introducing SMILE, a model-agnostic method that provides visual heatmaps to interpret textual influences, improving interpretability and reliability as measured by stability, accuracy, fidelity, and consistency metrics.
Despite recent advancements in Instruct-based Image Editing models for generating high-quality images, they are known as black boxes and a significant barrier to transparency and user trust. To solve this issue, we introduce SMILE (Statistical Model-agnostic Interpretability with Local Explanations), a novel model-agnostic for localized interpretability that provides a visual heatmap to clarify the textual elements' influence on image-generating models. We applied our method to various Instruction-based Image Editing models like Pix2Pix, Image2Image-turbo and Diffusers-Inpaint and showed how our model can improve interpretability and reliability. Also, we use stability, accuracy, fidelity, and consistency metrics to evaluate our method. These findings indicate the exciting potential of model-agnostic interpretability for reliability and trustworthiness in critical applications such as healthcare and autonomous driving while encouraging additional investigation into the significance of interpretability in enhancing dependable image editing models.