Locating and Editing Figure-Ground Organization in Vision Transformers
This research provides insights into the internal mechanisms of Vision Transformers for researchers studying model interpretability and perceptual organization, specifically how Gestalt principles are encoded.
This paper investigates how Vision Transformers resolve figure-ground organization, specifically the Gestalt prior of convexity, using synthetic dart shapes. They found that BEiT consistently favors convex completion and that this preference is governed by specific functional units within later transformer layers. By downscaling attention head L0H9, they were able to shift the model's decision boundary from convex to concave completion.
Vision Transformers must resolve figure-ground organization by choosing between completions driven by local geometric evidence and those favored by global organizational priors, giving rise to a characteristic perceptual ambiguity. We aim to locate where the canonical Gestalt prior convexity is realized within the internal components of BEiT. Using a controlled perceptual conflict based on synthetic shapes of darts, we systematically mask regions that equally admit either a concave completion or a convex completion. We show that BEiT reliably favors convex completion under this competition. Projecting internal activations into the model's discrete visual codebook space via logit attribution reveals that this preference is governed by identifiable functional units within transformer substructures. Specifically, we find that figure-ground organization is ambiguous through early and intermediate layers and resolves abruptly in later layers. By decomposing the direct effect of attention heads, we identify head L0H9 acting as an early seed, introducing a weak bias toward convexity. Downscaling this single attention head shifts the distributional mass of the perceptual conflict across a continuous decision boundary, allowing concave evidence to guide completion.