Selective Probabilistic Classifier Based on Hypothesis Testing
This addresses safety-critical applications where low false positives are essential, though it is an incremental improvement over existing threshold-based methods.
The paper tackles the problem of classifiers failing under the Closed-World Assumption by proposing a rejection method based on hypothesis testing with probabilistic networks, which achieves a lower False Positive Ratio compared to the Softmax Response method.
In this paper, we propose a simple yet effective method to deal with the violation of the Closed-World Assumption for a classifier. Previous works tend to apply a threshold either on the classification scores or the loss function to reject the inputs that violate the assumption. However, these methods cannot achieve the low False Positive Ratio (FPR) required in safety applications. The proposed method is a rejection option based on hypothesis testing with probabilistic networks. With probabilistic networks, it is possible to estimate the distribution of outcomes instead of a single output. By utilizing Z-test over the mean and standard deviation for each class, the proposed method can estimate the statistical significance of the network certainty and reject uncertain outputs. The proposed method was experimented on with different configurations of the COCO and CIFAR datasets. The performance of the proposed method is compared with the Softmax Response, which is a known top-performing method. It is shown that the proposed method can achieve a broader range of operation and cover a lower FPR than the alternative.