TaCOS: Task-Specific Camera Optimization with Simulation
This addresses the costly and isolated design process for cameras in perception tasks, offering a more efficient and automated solution for camera developers, though it is incremental as it builds on joint design methods.
The paper tackles the problem of designing cameras optimized for specific vision tasks by introducing an end-to-end optimization approach that co-designs cameras with tasks, resulting in cameras that outperform off-the-shelf options and reduce design time from 67 minutes to 2 minutes compared to state-of-the-art methods.
The performance of perception tasks is heavily influenced by imaging systems. However, designing cameras with high task performance is costly, requiring extensive camera knowledge and experimentation with physical hardware. Additionally, cameras and perception tasks are mostly designed in isolation, whereas recent methods that jointly design cameras and tasks have shown improved performance. Therefore, we present a novel end-to-end optimization approach that co-designs cameras with specific vision tasks. This method combines derivative-free and gradient-based optimizers to support both continuous and discrete camera parameters within manufacturing constraints. We leverage recent computer graphics techniques and physical camera characteristics to simulate the cameras in virtual environments, making the design process cost-effective. We validate our simulations against physical cameras and provide a procedurally generated virtual environment. Our experiments demonstrate that our method designs cameras that outperform common off-the-shelf options, and more efficiently compared to the state-of-the-art approach, requiring only 2 minutes to design a camera on an example experiment compared with 67 minutes for the competing method. Designed to support the development of cameras under manufacturing constraints, multiple cameras, and unconventional cameras, we believe this approach can advance the fully automated design of cameras.