Virtual Control Group: Measuring Hidden Performance Metrics
This work addresses a critical measurement challenge for financial integrity and cybersecurity domains, offering a novel estimation method.
The paper tackles the problem of estimating the false positive rate in financial integrity systems, where direct monitoring is impossible, by proposing a statistical method based on survey theory and causal inference, and introduces an outcome matching approach that outperforms other methods in some empirical cases.
Performance metrics measuring in Financial Integrity systems are crucial for maintaining an efficient and cost effective operation. An important performance metric is False Positive Rate. This metric cannot be directly monitored since we don't know for sure if a user is bad once blocked. We present a statistical method based on survey theory and causal inference methods to estimate the false positive rate of the system or a single blocking policy. We also suggest a new approach of outcome matching that in some cases including empirical data outperformed other commonly used methods. The approaches described in this paper can be applied in other Integrity domains such as Cyber Security.