Segment This Thing: Foveated Tokenization for Efficient Point-Prompted Segmentation
This work addresses efficiency challenges in segmentation for applications like augmented reality or robotics, though it is incremental as it builds on existing point-prompted segmentation methods.
The paper tackles the problem of efficient image segmentation from point prompts by introducing a foveated tokenization method that reduces computational cost without shrinking model size, achieving interactive frame rates on consumer hardware while remaining competitive on benchmarks.
This paper presents Segment This Thing (STT), a new efficient image segmentation model designed to produce a single segment given a single point prompt. Instead of following prior work and increasing efficiency by decreasing model size, we gain efficiency by foveating input images. Given an image and a point prompt, we extract a crop centered on the prompt and apply a novel variable-resolution patch tokenization in which patches are downsampled at a rate that increases with increased distance from the prompt. This approach yields far fewer image tokens than uniform patch tokenization. As a result we can drastically reduce the computational cost of segmentation without reducing model size. Furthermore, the foveation focuses the model on the region of interest, a potentially useful inductive bias. We show that our Segment This Thing model is more efficient than prior work while remaining competitive on segmentation benchmarks. It can easily run at interactive frame rates on consumer hardware and is thus a promising tool for augmented reality or robotics applications.