Data Distribution Valuation Using Generalized Bayesian Inference
This addresses the problem of quantifying data distribution values for applications such as annotator evaluation and data augmentation, representing a novel method for a known bottleneck.
The paper tackles the data distribution valuation problem by developing Generalized Bayes Valuation, a framework that uses generalized Bayesian inference with transferability-based loss functions to quantify data distribution values from samples. Experimental results confirm the framework's effectiveness and efficiency in real-world scenarios like annotator evaluation and data augmentation.
We investigate the data distribution valuation problem, which aims to quantify the values of data distributions from their samples. This is a recently proposed problem that is related to but different from classical data valuation and can be applied to various applications. For this problem, we develop a novel framework called Generalized Bayes Valuation that utilizes generalized Bayesian inference with a loss constructed from transferability measures. This framework allows us to solve, in a unified way, seemingly unrelated practical problems, such as annotator evaluation and data augmentation. Using the Bayesian principles, we further improve and enhance the applicability of our framework by extending it to the continuous data stream setting. Our experiment results confirm the effectiveness and efficiency of our framework in different real-world scenarios.