Is Data Shapley Not Better than Random in Data Selection? Ask NASH
For practitioners needing reliable data selection, this work provides a method to consistently outperform random selection, addressing a key limitation of Shapley-based approaches.
The paper addresses the inconsistent performance of Data Shapley for data selection, proposing NASH, a framework that decomposes the utility function into Shapley-informative components and aggregates them non-linearly, achieving substantial improvements in effectiveness with minimal runtime cost.
Data selection studies the problem of identifying high-quality subsets of training data. While some existing works have considered selecting the subset of data with top-$m$ Data Shapley or other semivalues as they account for the interaction among every subset of data, other works argue that Data Shapley can sometimes perform ineffectively in practice and select subsets that are no better than random. This raises the questions: (I) Are there certain "Shapley-informative" settings where Data Shapley consistently works well? (II) Can we strategically utilize these settings to select high-quality subsets consistently and efficiently? In this paper, we propose a novel data selection framework, NASH (Non-linear Aggregation of SHapley-informative components), which (I) decomposes the target utility function (e.g., validation accuracy) into simpler, Shapley-informative component functions, and selects data by optimizing an objective that (II) aggregates these components non-linearly. We demonstrate that NASH substantially boosts the effectiveness of Shapley/semivalue-based data selection with minimal additional runtime cost.