A case for data valuation transparency via DValCards
This addresses the problem of unreliable data valuation for ML practitioners and data market participants, highlighting ethical risks in incremental improvements to existing methods.
The paper demonstrates that data valuation metrics are biased and unstable under simple algorithmic choices, showing that pre-processing can drastically alter values, subsampling may increase class imbalance, and underrepresented groups may be undervalued. It proposes Data Valuation Cards (DValCards) to increase transparency and reduce misuse in applications like data pricing.
Following the rise in popularity of data-centric machine learning (ML), various data valuation methods have been proposed to quantify the contribution of each datapoint to desired ML model performance metrics (e.g., accuracy). Beyond the technical applications of data valuation methods (e.g., data cleaning, data acquisition, etc.), it has been suggested that within the context of data markets, data buyers might utilize such methods to fairly compensate data owners. Here we demonstrate that data valuation metrics are inherently biased and unstable under simple algorithmic design choices, resulting in both technical and ethical implications. By analyzing 9 tabular classification datasets and 6 data valuation methods, we illustrate how (1) common and inexpensive data pre-processing techniques can drastically alter estimated data values; (2) subsampling via data valuation metrics may increase class imbalance; and (3) data valuation metrics may undervalue underrepresented group data. Consequently, we argue in favor of increased transparency associated with data valuation in-the-wild and introduce the novel Data Valuation Cards (DValCards) framework towards this aim. The proliferation of DValCards will reduce misuse of data valuation metrics, including in data pricing, and build trust in responsible ML systems.