Taking Class Imbalance Into Account in Open Set Recognition Evaluation
This work provides incremental improvements for researchers in machine learning by focusing on evaluation methodology rather than novel algorithms.
The paper addresses the evaluation of Open Set Recognition methods by analyzing the impact of class imbalance between known and unknown samples, resulting in a set of guidelines for better evaluation practices.
In recent years Deep Neural Network-based systems are not only increasing in popularity but also receive growing user trust. However, due to the closed-world assumption of such systems, they cannot recognize samples from unknown classes and often induce an incorrect label with high confidence. Presented work looks at the evaluation of methods for Open Set Recognition, focusing on the impact of class imbalance, especially in the dichotomy between known and unknown samples. As an outcome of problem analysis, we present a set of guidelines for evaluation of methods in this field.