TiC-LM: A Web-Scale Benchmark for Time-Continual LLM Pretraining
This addresses the challenge of keeping LLMs up-to-date efficiently, though it is incremental as it builds on existing continual learning methods.
The authors tackled the problem of large language models becoming outdated by introducing a web-scale dataset for time-continual pretraining and evaluating update methods, finding that autoregressive meta-schedules with fixed-ratio replay achieve comparable performance to full retraining with 2.6x less computation.
Large Language Models (LLMs) trained on historical web data inevitably become outdated. We investigate evaluation strategies and update methods for LLMs as new data becomes available. We introduce a web-scale dataset for time-continual pretraining of LLMs derived from 114 dumps of Common Crawl (CC) - orders of magnitude larger than previous continual language modeling benchmarks. We also design time-stratified evaluations across both general CC data and specific domains (Wikipedia, StackExchange, and code documentation) to assess how well various continual learning methods adapt to new data while retaining past knowledge. Our findings demonstrate that, on general CC data, autoregressive meta-schedules combined with a fixed-ratio replay of older data can achieve comparable held-out loss to re-training from scratch, while requiring significantly less computation (2.6x). However, the optimal balance between incorporating new data and replaying old data differs as replay is crucial to avoid forgetting on generic web data but less so on specific domains.