Beyond URLs: Metadata Diversity and Position for Efficient LLM Pretraining
This work addresses the challenge of efficient LLM pretraining for AI researchers and practitioners, offering incremental improvements over prior methods focused on URLs.
The study tackled the problem of accelerating LLM pretraining by exploring diverse metadata types beyond URLs, finding that fine-grained quality indicators and metadata appending as an auxiliary task can improve training efficiency, with learnable meta-tokens recovering part of the speedup.
Incorporating metadata in Large Language Models (LLMs) pretraining has recently emerged as a promising approach to accelerate training. However prior work highlighted only one useful signal-URLs, leaving open the question of whether other forms of metadata could yield greater benefits. In this study, we investigate a wider range of metadata types and find other types of metadata, such as fine-grained indicators of document quality that can also accelerate pretraining when prepended. We identify a common feature among effective metadata: they encode information at a finer granularity. We further introduce metadata appending as a means of improving training efficiency, where predicting an appropriate metadata as auxiliary task can help speed up pretraining. In addition, learnable meta-tokens trained with masked loss can recover part of the speedup by inducing quality-aware latent structure. Using probing, we analyze latent representations to understand how metadata shapes learning. Together, these results yield practical guidelines for integrating metadata to improve both the efficiency and effectiveness of LLM pretraining.