JOCA: Task-Driven Joint Optimisation of Camera Hardware and Adaptive Camera Control Algorithms
This work addresses the challenge of designing camera systems for better vision task performance, but it is incremental as it builds on prior co-design methods by adding adaptive control.
The paper tackles the problem of improving perception task performance by jointly optimizing camera hardware and adaptive control algorithms, rather than separately, and shows that this approach outperforms baselines, especially in low-light and fast-motion conditions.
The quality of captured images strongly influences the performance of downstream perception tasks. Recent works on co-designing camera systems with perception tasks have shown improved task performance. However, most prior approaches focus on optimising fixed camera parameters set at manufacturing, while many parameters, such as exposure settings, require adaptive control at runtime. This paper introduces a method that jointly optimises camera hardware and adaptive camera control algorithms with downstream vision tasks. We present a unified optimisation framework that integrates gradient-based and derivative-free methods, enabling support for both continuous and discrete parameters, non-differentiable image formation processes, and neural network-based adaptive control algorithms. To address non-differentiable effects such as motion blur, we propose DF-Grad, a hybrid optimisation strategy that trains adaptive control networks using signals from a derivative-free optimiser alongside unsupervised task-driven learning. Experiments show that our method outperforms baselines that optimise static and dynamic parameters separately, particularly under challenging conditions such as low light and fast motion. These results demonstrate that jointly optimising hardware parameters and adaptive control algorithms improves perception performance and provides a unified approach to task-driven camera system design.