Learning to Sense for Driving: Joint Optics-Sensor-Model Co-Design for Semantic Segmentation
This work addresses the need for efficient and reliable perception in autonomous systems by enabling full-stack optimization, though it builds incrementally on prior methods like DeepLens.
The paper tackles the problem of decoupled camera design in autonomous driving by introducing a task-driven co-design framework that jointly optimizes optics, sensor modeling, and semantic segmentation networks, achieving consistent mIoU improvements on KITTI-360 with a compact ~1M-parameter model running at ~28 FPS.
Traditional autonomous driving pipelines decouple camera design from downstream perception, relying on fixed optics and handcrafted ISPs that prioritize human viewable imagery rather than machine semantics. This separation discards information during demosaicing, denoising, or quantization, while forcing models to adapt to sensor artifacts. We present a task-driven co-design framework that unifies optics, sensor modeling, and lightweight semantic segmentation networks into a single end-to-end RAW-to-task pipeline. Building on DeepLens[19], our system integrates realistic cellphone-scale lens models, learnable color filter arrays, Poisson-Gaussian noise processes, and quantization, all optimized directly for segmentation objectives. Evaluations on KITTI-360 show consistent mIoU improvements over fixed pipelines, with optics modeling and CFA learning providing the largest gains, especially for thin or low-light-sensitive classes. Importantly, these robustness gains are achieved with a compact ~1M-parameter model running at ~28 FPS, demonstrating edge deployability. Visual and quantitative analyses further highlight how co-designed sensors adapt acquisition to semantic structure, sharpening boundaries and maintaining accuracy under blur, noise, and low bit-depth. Together, these findings establish full-stack co-optimization of optics, sensors, and networks as a principled path toward efficient, reliable, and deployable perception in autonomous systems.