PicoSAM3: Real-Time In-Sensor Region-of-Interest Segmentation
This work addresses the problem of latency-sensitive and privacy-aware segmentation for smart glasses and IoT devices, representing an incremental improvement with specific optimizations for edge deployment.
The paper tackles real-time, on-device segmentation for edge and in-sensor applications by introducing PicoSAM3, a lightweight model with 1.3M parameters that achieves 65.45% mIoU on COCO and 64.01% mIoU on LVIS, outperforming baselines at similar complexity, and enables 11.82 ms latency on the Sony IMX500 sensor with INT8 quantization.
Real-time, on-device segmentation is critical for latency-sensitive and privacy-aware applications such as smart glasses and Internet-of-Things devices. We introduce PicoSAM3, a lightweight promptable visual segmentation model optimized for edge and in-sensor execution, including deployment on the Sony IMX500 vision sensor. PicoSAM3 has 1.3 M parameters and combines a dense CNN architecture with region of interest prompt encoding, Efficient Channel Attention, and knowledge distillation from SAM2 and SAM3. On COCO and LVIS, PicoSAM3 achieves 65.45% and 64.01% mIoU, respectively, outperforming existing SAM-based and edge-oriented baselines at similar or lower complexity. The INT8 quantized model preserves accuracy with negligible degradation while enabling real-time in-sensor inference at 11.82 ms latency on the IMX500, fully complying with its memory and operator constraints. Ablation studies show that distillation from large SAM models yields up to +14.5% mIoU improvement over supervised training and demonstrate that high-quality, spatially flexible promptable segmentation is feasible directly at the sensor level.