CLUSTSEG: Clustering for Universal Segmentation
This provides a general solution for various segmentation problems in computer vision, though it appears incremental as it builds on existing transformer and clustering ideas.
The paper tackles multiple image segmentation tasks (superpixel, semantic, instance, and panoptic) with a unified neural clustering framework called CLUSTSEG, achieving superior results across these tasks.
We present CLUSTSEG, a general, transformer-based framework that tackles different image segmentation tasks (i.e., superpixel, semantic, instance, and panoptic) through a unified neural clustering scheme. Regarding queries as cluster centers, CLUSTSEG is innovative in two aspects:1) cluster centers are initialized in heterogeneous ways so as to pointedly address task-specific demands (e.g., instance- or category-level distinctiveness), yet without modifying the architecture; and 2) pixel-cluster assignment, formalized in a cross-attention fashion, is alternated with cluster center update, yet without learning additional parameters. These innovations closely link CLUSTSEG to EM clustering and make it a transparent and powerful framework that yields superior results across the above segmentation tasks.