Moondream Segmentation: From Words to Masks
This addresses the problem of precise object segmentation from natural language queries for computer vision applications, representing an incremental improvement with new evaluation data.
The paper tackles referring image segmentation by extending a vision-language model to generate detailed masks from images and text descriptions, achieving a cIoU of 80.2% on RefCOCO and 62.6% mIoU on LVIS.
We present Moondream Segmentation, a referring image segmentation extension of Moondream 3, a vision-language model. Given an image and a referring expression, the model autoregressively decodes a vector path and iteratively refines the rasterized mask into a final detailed mask. We introduce a reinforcement learning stage that resolves ambiguity in the supervised signal by directly optimizing mask quality. Rollouts from this stage produce coarse-to-ground-truth targets for the refiner. To mitigate evaluation noise from polygon annotations, we release RefCOCO-M, a cleaned RefCOCO validation split with boundary-accurate masks. Moondream Segmentation achieves a cIoU of 80.2% on RefCOCO (val) and 62.6% mIoU on LVIS (val).