Prediction-powered estimators for finite population statistics in highly imbalanced textual data: Public hate crime estimation
This method addresses the challenge of estimating hate crime statistics from police reports for policymakers, but it is incremental as it adapts existing techniques to a specific domain.
The authors tackled the problem of estimating population parameters in finite text populations where manual labeling is costly, by combining transformer encoder predictions with survey sampling estimators, and demonstrated its application to Swedish hate crime statistics, showing it can provide efficient estimates with reduced annotation time.
Estimating population parameters in finite populations of text documents can be challenging when obtaining the labels for the target variable requires manual annotation. To address this problem, we combine predictions from a transformer encoder neural network with well-established survey sampling estimators using the model predictions as an auxiliary variable. The applicability is demonstrated in Swedish hate crime statistics based on Swedish police reports. Estimates of the yearly number of hate crimes and the police's under-reporting are derived using the Hansen-Hurwitz estimator, difference estimation, and stratified random sampling estimation. We conclude that if labeled training data is available, the proposed method can provide very efficient estimates with reduced time spent on manual annotation.