PicoSAM2: Low-Latency Segmentation In-Sensor for Edge Vision Applications
This enables privacy-preserving, low-latency segmentation for edge devices like smart glasses and IoT, though it is incremental as it builds on existing models like SAM2.
The paper tackled the problem of real-time, on-device segmentation for edge vision applications by introducing PicoSAM2, a lightweight model that achieved 51.9% mIoU on COCO and 44.9% mIoU on LVIS, with a quantized version running at 14.3 ms on the Sony IMX500 sensor.
Real-time, on-device segmentation is critical for latency-sensitive and privacy-aware applications like smart glasses and IoT devices. We introduce PicoSAM2, a lightweight (1.3M parameters, 336M MACs) promptable segmentation model optimized for edge and in-sensor execution, including the Sony IMX500. It builds on a depthwise separable U-Net, with knowledge distillation and fixed-point prompt encoding to learn from the Segment Anything Model 2 (SAM2). On COCO and LVIS, it achieves 51.9% and 44.9% mIoU, respectively. The quantized model (1.22MB) runs at 14.3 ms on the IMX500-achieving 86 MACs/cycle, making it the only model meeting both memory and compute constraints for in-sensor deployment. Distillation boosts LVIS performance by +3.5% mIoU and +5.1% mAP. These results demonstrate that efficient, promptable segmentation is feasible directly on-camera, enabling privacy-preserving vision without cloud or host processing.