Beyond Bayer: Task-Optimal Sensor Co-Design for Robust Autonomous-Driving Segmentation
For autonomous driving perception, this work provides a practical recipe for sensor design that yields consistent, model-agnostic improvements in segmentation robustness.
This paper investigates sensor-level optimization for autonomous driving segmentation by learning camera parameters such as color filter array (CFA) weights, point-spread function (PSF), and noise characteristics. Learning CFA weights improves mIoU by +0.017 on KITTI-360 and +0.023 on ACDC, while PSF co-design is net-negative (-0.020 mIoU on KITTI-360), and the gains are model-agnostic.
Robust perception underpins autonomous driving, and most recent progress comes from scaling the model-larger backbones, foundation models, and cooperative multi-agent fusion. We pursue a complementary, upstream question: what should the camera itself measure? Using a differentiable RAW-to-task pipeline, we decompose which sensor degrees of freedom benefit dense prediction. Learning the spectral colour-filter-array (CFA) weights is the dominant lever, improving mIoU by +0.017 (KITTI-360) and +0.023 (ACDC) over a fixed camera. In contrast, point-spread-function (optics) co-design is net-negative (-0.020 mIoU on KITTI-360) - a consequence of the data-processing inequality, which also bounds the task information that any downstream model, however large or cooperative, can recover. Noise co-optimisation is marginal, and counter to intuition enlarging the CFA tile beyond 2x2 consistently hurts, as the filters are confined to the rank three sRGB input. Because the intervention is at the sensor, the gains are model-agnostic; we validate robustness on ACDC's fog, night, rain, and snow, and conclude with a simple recipe: learn the 2x2 CFA weights and keep an identity PSF.