Do we really have to filter out random noise in pre-training data for language models?
This research addresses a problem for natural language processing and multimodal model developers, providing insights into the effects of random noise in pre-training data and a potential solution to mitigate its adverse effects.
The study investigated the impact of random noise in pre-training data on language models, finding that the increase in next-token prediction loss was lower than expected, but downstream performance may still be degraded, with experiments showing the effectiveness of a novel Local Gradient Matching loss. The model's performance was tested on 8 language and 14 vision benchmarks.
Web-scale pre-training datasets are the cornerstone of LLMs' success. However, text data curated from the Internet inevitably contains random noise caused by decoding errors or unregulated web content. In contrast to previous works that focus on low quality or synthetic data, our study \textbf{provides the first systematic investigation of such random noise through a cohesive ``What-Why-How'' framework.} Surprisingly, we observed that the resulting increase in the loss of next-token prediction (NTP) was significantly lower than the proportion of random noise even when the model was scaled up to 2.7B. We provide a theoretical justification for this phenomenon, which also elucidates the success of multilingual models and can be applied to multimodal models. On the other hand, experiments show that the model's performance in downstream tasks is not based solely on the NTP loss, which means that random noise may result in degraded downstream performance. To address the potential adverse effects, we introduce a novel plug-and-play Local Gradient Matching loss, which explicitly enhances the denoising capability of the downstream task head by aligning the gradient of normal and perturbed features without requiring knowledge of the model's parameters. Additional experiments on 8 language and 14 vision benchmarks further validate its effectiveness.