Laplace Sample Information: Data Informativeness Through a Bayesian Lens
This addresses the need for efficient sample selection in deep learning to improve model performance, though it is incremental as it builds on existing Bayesian and information-theoretic methods.
The paper tackles the problem of estimating sample informativeness in datasets by proposing Laplace Sample Information (LSI), a measure based on Bayesian approximation and KL divergence, which effectively orders data by typicality, detects mislabeled samples, and assesses dataset difficulty across image and text data in supervised and unsupervised settings.
Accurately estimating the informativeness of individual samples in a dataset is an important objective in deep learning, as it can guide sample selection, which can improve model efficiency and accuracy by removing redundant or potentially harmful samples. We propose Laplace Sample Information (LSI) measure of sample informativeness grounded in information theory widely applicable across model architectures and learning settings. LSI leverages a Bayesian approximation to the weight posterior and the KL divergence to measure the change in the parameter distribution induced by a sample of interest from the dataset. We experimentally show that LSI is effective in ordering the data with respect to typicality, detecting mislabeled samples, measuring class-wise informativeness, and assessing dataset difficulty. We demonstrate these capabilities of LSI on image and text data in supervised and unsupervised settings. Moreover, we show that LSI can be computed efficiently through probes and transfers well to the training of large models.