Tokenizing Semantic Segmentation with RLE
This work addresses segmentation tasks for computer vision applications, but it is incremental as it builds on existing methods like Pix2Seq.
The paper tackles semantic segmentation in images and videos by using language modeling to output masks as discrete tokens via run-length encoding, achieving competitive results with state-of-the-art methods despite computational limitations.
This paper presents a new unified approach to semantic segmentation in both images and videos by using language modeling to output the masks as sequences of discrete tokens. We use run length encoding (RLE) to discretize the segmentation masks and then train a modified version of Pix2Seq \cite{p2s} to output these RLE tokens through autoregression. We propose novel tokenization strategies to compress the length of the token sequence to make it practicable to extend this approach to videos. We also show how instance information can be incorporated into the tokenization process to perform panoptic segmentation. We evaluate our proposed models on two datasets to show that they are competitive with the state of the art in spite of being bottlenecked by our limited computational resources.