R-divergence for Estimating Model-oriented Distribution Discrepancy
This addresses the challenge of non-IID data in machine learning, providing a tool for model evaluation and robust training, though it appears incremental as it builds on existing divergence concepts.
The paper tackles the problem of assessing whether two datasets have identical probability distributions by introducing R-divergence, which estimates model-oriented distribution discrepancies and achieves state-of-the-art performance in various tasks.
Real-life data are often non-IID due to complex distributions and interactions, and the sensitivity to the distribution of samples can differ among learning models. Accordingly, a key question for any supervised or unsupervised model is whether the probability distributions of two given datasets can be considered identical. To address this question, we introduce R-divergence, designed to assess model-oriented distribution discrepancies. The core insight is that two distributions are likely identical if their optimal hypothesis yields the same expected risk for each distribution. To estimate the distribution discrepancy between two datasets, R-divergence learns a minimum hypothesis on the mixed data and then gauges the empirical risk difference between them. We evaluate the test power across various unsupervised and supervised tasks and find that R-divergence achieves state-of-the-art performance. To demonstrate the practicality of R-divergence, we employ R-divergence to train robust neural networks on samples with noisy labels.