AVOIDDS: Aircraft Vision-based Intruder Detection Dataset and Simulator
This provides a domain-specific benchmark for researchers working on robust vision systems in aviation safety, though it is incremental as it focuses on creating a new dataset and tool rather than a novel method.
The authors tackled the need for robust machine learning benchmarks in safety-critical applications by introducing AVOIDDS, a realistic object detection dataset and simulator for aircraft detect-and-avoid, resulting in a dataset of 72,000 photorealistic images and an integrated simulator for evaluation.
Designing robust machine learning systems remains an open problem, and there is a need for benchmark problems that cover both environmental changes and evaluation on a downstream task. In this work, we introduce AVOIDDS, a realistic object detection benchmark for the vision-based aircraft detect-and-avoid problem. We provide a labeled dataset consisting of 72,000 photorealistic images of intruder aircraft with various lighting conditions, weather conditions, relative geometries, and geographic locations. We also provide an interface that evaluates trained models on slices of this dataset to identify changes in performance with respect to changing environmental conditions. Finally, we implement a fully-integrated, closed-loop simulator of the vision-based detect-and-avoid problem to evaluate trained models with respect to the downstream collision avoidance task. This benchmark will enable further research in the design of robust machine learning systems for use in safety-critical applications. The AVOIDDS dataset and code are publicly available at https://purl.stanford.edu/hj293cv5980 and https://github.com/sisl/VisionBasedAircraftDAA respectively.