Weakly Supervised Incremental Segmentation via Semantic Anchors and Spatial Arbitration
For researchers in incremental learning and semantic segmentation, this work addresses the critical issue of feature drift under weak supervision, offering a robust solution that outperforms existing methods.
The paper tackles the problem of weakly supervised incremental semantic segmentation, where noisy supervision causes feature drift and overwriting of old classes. The proposed SASA method uses semantic anchors and spatial arbitration to stabilize representations and improve supervision reliability, achieving state-of-the-art performance on standard benchmarks, especially in multi-step incremental settings.
Weakly Incremental Learning for Semantic Segmentation (WILSS) suffers from the continuous introduction of noisy supervision, which progressively corrupts class-level representations, leading to severe feature drift and semantic corruption, thereby causing newly learned classes to overwrite old ones. To address these issues, we propose a drift-resilient WILSS approach, named SASA, designed to stabilize semantic learning via Semantic Anchors and Spatial Arbitration. Specifically, at the representation level, we introduce semantic anchors of learnable tokens as rigid class-level references to preserve long-term semantic identity. Complementary to this, an elastic residual adaptation facilitates controlled, instance-specific refinement, ensuring a stable yet flexible learning trajectory. At the supervision level, we develop a Spatial Label Arbitration mechanism that performs geometry-aware decisions to directly filter unreliable signals and enforce a strict "one object, one class" constraint. By synergistically stabilizing representations and improving supervision reliability, SASA effectively mitigates feature drift under weak supervision. Extensive experiments on standard benchmarks demonstrate that our approach consistently outperforms existing state-of-the-art methods, particularly in challenging multi-step incremental settings. The code is available at https://github.com/ZhonggaiWang/SASA.