Data Reliability Scoring
This addresses the challenge of evaluating data quality from strategic sources for researchers and practitioners in data science and machine learning, offering a novel benchmark for reliability assessment.
The paper tackles the problem of assessing dataset reliability without ground truth by introducing the Gram determinant score, which measures deviations from true data using empirical distributions and experiment outcomes, and demonstrates its effectiveness across synthetic and real datasets.
How can we assess the reliability of a dataset without access to ground truth? We introduce the problem of reliability scoring for datasets collected from potentially strategic sources. The true data are unobserved, but we see outcomes of an unknown statistical experiment that depends on them. To benchmark reliability, we define ground-truth-based orderings that capture how much reported data deviate from the truth. We then propose the Gram determinant score, which measures the volume spanned by vectors describing the empirical distribution of the observed data and experiment outcomes. We show that this score preserves several ground-truth based reliability orderings and, uniquely up to scaling, yields the same reliability ranking of datasets regardless of the experiment -- a property we term experiment agnosticism. Experiments on synthetic noise models, CIFAR-10 embeddings, and real employment data demonstrate that the Gram determinant score effectively captures data quality across diverse observation processes.