Using simulation to quantify the performance of automotive perception systems
This work addresses the challenge of testing automotive perception systems under varied conditions for engineers, but it is incremental as it applies existing simulation methods to new data.
The authors tackled the problem of evaluating automotive perception systems by using simulation to quantify object detection performance across different camera resolutions and lighting conditions, reporting a trend between resolution and performance and a significant degradation at night.
The design and evaluation of complex systems can benefit from a software simulation - sometimes called a digital twin. The simulation can be used to characterize system performance or to test its performance under conditions that are difficult to measure (e.g., nighttime for automotive perception systems). We describe the image system simulation software tools that we use to evaluate the performance of image systems for object (automobile) detection. We describe experiments with 13 different cameras with a variety of optics and pixel sizes. To measure the impact of camera spatial resolution, we designed a collection of driving scenes that had cars at many different distances. We quantified system performance by measuring average precision and we report a trend relating system resolution and object detection performance. We also quantified the large performance degradation under nighttime conditions, compared to daytime, for all cameras and a COCO pre-trained network.