What Pixels Are Enough? SEAMS: Sufficiency Saliency via MSE-Preservation Soft-Masks
Provides a simple, general-purpose sufficiency saliency method for interpreting vision models, useful for practitioners needing faithful explanations without complex setups.
SEAMS introduces a sufficiency-based saliency method that optimizes a soft mask to preserve a model's output, achieving compact and interpretable masks without auxiliary data or architecture-specific mechanisms. It demonstrates competitive performance on insertion/deletion benchmarks and reveals architecture-dependent visual explanations.
Saliency maps are most useful when they identify the image regions that are sufficient to preserve a model's behaviour. We introduce SEAMS, a sufficiency-based saliency method that directly optimises a soft mask using a preservation objective. Given a frozen differentiable model output, such as a class probability, CLS embedding, or token representation, SEAMS searches for a compact mask that preserves the selected output. The approach relies on a simple optimisation framework based on soft masks, a learnable budget, and a three-way image composite generated entirely from the query image. As a result, it requires no auxiliary distractor dataset, architecture-specific attribution mechanism, or differentiable top-k relaxation. Experiments with frozen ViT-S/16 and ConvNeXt models show that the same optimisation pipeline can generate object-level, class-conditioned, and token-level explanations by changing only the preserved target. The resulting masks are compact, interpretable, stable across random initialisations, and competitive on insertion and deletion benchmarks. Our results also indicate that different architectures often rely on different sufficient evidence while achieving similar preservation fidelity, highlighting the architecture-dependent nature of visual explanations.