Robustness of Probabilistic Models to Low-Quality Data: A Multi-Perspective Analysis
This addresses the problem of model reliability under data corruption for AI practitioners, providing insights into robustness variations across model types.
The paper systematically compares the robustness of probabilistic models to low-quality data, finding that autoregressive language models like GPT-2 are resilient (test NLL increases from 2.87 to 3.59 with 50% token corruption), while class-conditional diffusion models degrade severely (image-label consistency drops by 56.81%) and classifiers show moderate effects.
A systematic, comparative investigation into the effects of low-quality data reveals a stark spectrum of robustness across modern probabilistic models. We find that autoregressive language models, from token prediction to sequence-to-sequence tasks, are remarkably resilient (for GPT-2, test NLL increases modestly from 2.87 to 3.59 despite 50% token corruption). By contrast, under the same levels of data corruption, class-conditional diffusion models degrade catastrophically (image-label consistency plummets by 56.81% relative to baseline), while classifiers show a moderate impact that diminishes with dataset scale. To explain these discrepancies, we analyze the results through a multi-perspective lens, integrating information theory, PAC learning, and gradient dynamics. These analyses suggest that robustness is heavily influenced by two key principles: the richness of conditioning information, which constrains the learning problem, and the absolute information content of the training data, which allows the signal from correct information to dominate statistical noise.