RICO: Two Realistic Benchmarks and an In-Depth Analysis for Incremental Learning in Object Detection
This work addresses the gap in realistic evaluation for incremental learning in object detection, which is crucial for real-world applications like autonomous driving, though it is incremental as it focuses on benchmarking rather than proposing new methods.
The paper tackles the problem of evaluating incremental learning in object detection by introducing two realistic benchmarks, RICO, and finds that all existing methods underperform in adaptability and retention, with replaying a small amount of previous data outperforming them but still falling short of individual training.
Incremental Learning (IL) trains models sequentially on new data without full retraining, offering privacy, efficiency, and scalability. IL must balance adaptability to new data with retention of old knowledge. However, evaluations often rely on synthetic, simplified benchmarks, obscuring real-world IL performance. To address this, we introduce two Realistic Incremental Object Detection Benchmarks (RICO): Domain RICO (D-RICO) features domain shifts with a fixed class set, and Expanding-Classes RICO (EC-RICO) integrates new domains and classes per IL step. Built from 14 diverse datasets covering real and synthetic domains, varying conditions (e.g., weather, time of day), camera sensors, perspectives, and labeling policies, both benchmarks capture challenges absent in existing evaluations. Our experiments show that all IL methods underperform in adaptability and retention, while replaying a small amount of previous data already outperforms all methods. However, individual training on the data remains superior. We heuristically attribute this gap to weak teachers in distillation, single models' inability to manage diverse tasks, and insufficient plasticity. Our code will be made publicly available.