Improved Error Bounds Based on Worst Likely Assignments
This work addresses the challenge of reliable classifier validation for researchers and practitioners dealing with small datasets, though it is incremental as it builds on existing worst likely assignment methods.
The paper tackles the problem of validating classifiers with limited training data by improving error bounds based on worst likely assignments, resulting in enhanced bounds particularly for accurate classifiers.
Error bounds based on worst likely assignments use permutation tests to validate classifiers. Worst likely assignments can produce effective bounds even for data sets with 100 or fewer training examples. This paper introduces a statistic for use in the permutation tests of worst likely assignments that improves error bounds, especially for accurate classifiers, which are typically the classifiers of interest.