Tradeoffs in Streaming Binary Classification under Limited Inspection Resources
This work addresses the operational challenge of efficiently allocating inspection resources for institutions using ML for fraud detection and similar tasks, presenting an incremental analysis of threshold-based methods.
The paper tackles the problem of selecting suspicious events for manual inspection under limited capacity in streaming binary classification, modeling event arrivals as a non-homogeneous Poisson process and analytically characterizing the tradeoff between detection rate and inspection capacity, with empirical validation on a fraud detection dataset showing how class imbalance and classifier choice affect this tradeoff.
Institutions are increasingly relying on machine learning models to identify and alert on abnormal events, such as fraud, cyber attacks and system failures. These alerts often need to be manually investigated by specialists. Given the operational cost of manual inspections, the suspicious events are selected by alerting systems with carefully designed thresholds. In this paper, we consider an imbalanced binary classification problem, where events arrive sequentially and only a limited number of suspicious events can be inspected. We model the event arrivals as a non-homogeneous Poisson process, and compare various suspicious event selection methods including those based on static and adaptive thresholds. For each method, we analytically characterize the tradeoff between the minority-class detection rate and the inspection capacity as a function of the data class imbalance and the classifier confidence score densities. We implement the selection methods on a real public fraud detection dataset and compare the empirical results with analytical bounds. Finally, we investigate how class imbalance and the choice of classifier impact the tradeoff.