Buying Data of Unknown Quality: Fisher Information Procurement Auctions
For data market designers and buyers, this work provides theoretically grounded mechanisms to procure data from competing providers with private quality, addressing a key bottleneck in data acquisition.
This paper designs procurement mechanisms for buying data from providers with unknown quality, enabling a buyer to estimate a parameter while minimizing cost. The proposed mechanisms achieve truthful cost reports and asymptotically truthful quality reports, with the second-score mechanism selecting providers based on a cost-per-information score.
We study statistical parameter estimation in the setting of data markets. A buyer seeks to estimate a parameter based on samples that can be purchased from competing providers that differ in their data quality and provision costs. When quality is known ex ante, we define a cost-per-information score that summarizes each provider's provision cost per unit of information about the buyer's estimation objective. We describe second-score procurement mechanism that ranks providers by this score, and endogenously chooses both a provider and a sample size while making truthful cost reports optimal. We then turn to the more realistic setting where data quality is private, and can only be indirectly observed via the delivered data. In this setting, we propose a simple mechanism that augments the second-score rule with a lenient ex post statistical test of the reported quality. We prove that under mild conditions, there exists an equilibrium in which sellers report costs truthfully and report quality up to deviations that vanish as the procured sample size grows. Our analysis highlights how the choice of verification test and the buyer's accuracy-cost tradeoff jointly shape participation and misreporting incentives in data markets.