Extrapolating Expected Accuracies for Large Multi-Class Problems
This addresses the challenge of scalability in multi-class classification for researchers and practitioners, though it is incremental as it builds on existing assumptions and methods.
The paper tackles the problem of predicting classifier accuracy as the number of classes increases in multi-class classification, showing that the expected accuracy can be estimated as a moment of a distribution, and demonstrates this with an unbiased method on a facial recognition example.
The difficulty of multi-class classification generally increases with the number of classes. Using data from a subset of the classes, can we predict how well a classifier will scale with an increased number of classes? Under the assumptions that the classes are sampled identically and independently from a population, and that the classifier is based on independently learned scoring functions, we show that the expected accuracy when the classifier is trained on k classes is the (k-1)st moment of a certain distribution that can be estimated from data. We present an unbiased estimation method based on the theory, and demonstrate its application on a facial recognition example.