Photorealistic Inpainting for Perturbation-based Explanations in Ecological Monitoring
This addresses the need for trustworthy AI deployment in ecology by providing domain-relevant insights for expert validation, though it is incremental as it builds on existing perturbation and inpainting methods.
The paper tackles the problem of opaque predictions in automated ecological monitoring by developing an inpainting-guided perturbation-based explanation technique that produces photorealistic, mask-localized edits to reveal fine-grained morphological cues driving predictions, such as in harbor seal detection, with results assessed through metrics like flip rate and confidence drop.
Ecological monitoring is increasingly automated by vision models, yet opaque predictions limit trust and field adoption. We present an inpainting-guided, perturbation-based explanation technique that produces photorealistic, mask-localized edits that preserve scene context. Unlike masking or blurring, these edits stay in-distribution and reveal which fine-grained morphological cues drive predictions in tasks such as species recognition and trait attribution. We demonstrate the approach on a YOLOv9 detector fine-tuned for harbor seal detection in Glacier Bay drone imagery, using Segment-Anything-Model-refined masks to support two interventions: (i) object removal/replacement (e.g., replacing seals with plausible ice/water or boats) and (ii) background replacement with original animals composited onto new scenes. Explanations are assessed by re-scoring perturbed images (flip rate, confidence drop) and by expert review for ecological plausibility and interpretability. The resulting explanations localize diagnostic structures, avoid deletion artifacts common to traditional perturbations, and yield domain-relevant insights that support expert validation and more trustworthy deployment of AI in ecology.