Surrogate Representation Inference for Noisy Text and Image Annotations
This addresses a key issue for researchers relying on automated annotations in fields like social sciences or computer vision, offering a method to improve statistical reliability, though it appears incremental by building on existing correction approaches.
The paper tackles the problem of bias and large standard errors in statistical analysis when using machine learning models or LLMs to annotate unstructured data like text or images, introducing Surrogate Representation Inference (SRI) to reduce standard errors by over 50% in moderate accuracy scenarios and provide valid inference despite non-differential measurement errors in human annotations.
As researchers increasingly rely on machine learning models and LLMs to annotate unstructured data, such as texts or images, various approaches have been proposed to correct bias in downstream statistical analysis. However, existing methods tend to yield large standard errors and require some error-free human annotation. In this paper, I introduce Surrogate Representation Inference (SRI), which assumes that unstructured data fully mediate the relationship between human annotations and structured variables. The assumption is guaranteed by design provided that human coders rely only on unstructured data for annotation. Under this setting, I propose a neural network architecture that learns a low-dimensional representation of unstructured data such that the surrogate assumption remains to be satisfied. When multiple human annotations are available, SRI can further correct non-differential measurement errors that may exist in human annotations. Focusing on text-as-outcome settings, I formally establish the identification conditions and semiparametric efficient estimation strategies that enable learning and leveraging such a low-dimensional representation. Simulation studies and a real-world application demonstrate that SRI reduces standard errors by over 50% when machine learning prediction accuracy is moderate and provides valid inference even when human annotations contain non-differential measurement errors.