How many classes do we need to see for novel class discovery?
This work addresses a fundamental question in machine learning for practitioners dealing with evolving data, though it is incremental as it builds on existing class discovery research.
The authors tackled the problem of understanding when novel class discovery is more likely to succeed by proposing a controlled experimental framework using the dSprites dataset. Their results show that the benefit of increasing known classes reaches a saturation point, with discovery performance plateauing beyond that.
Novel class discovery is essential for ML models to adapt to evolving real-world data, with applications ranging from scientific discovery to robotics. However, these datasets contain complex and entangled factors of variation, making a systematic study of class discovery difficult. As a result, many fundamental questions are yet to be answered on why and when new class discoveries are more likely to be successful. To address this, we propose a simple controlled experimental framework using the dSprites dataset with procedurally generated modifying factors. This allows us to investigate what influences successful class discovery. In particular, we study the relationship between the number of known/unknown classes and discovery performance, as well as the impact of known class 'coverage' on discovering new classes. Our empirical results indicate that the benefit of the number of known classes reaches a saturation point beyond which discovery performance plateaus. The pattern of diminishing return across different settings provides an insight for cost-benefit analysis for practitioners and a starting point for more rigorous future research of class discovery on complex real-world datasets.