Non-Vacuous Generalization Bounds for Large Language Models
This addresses the theoretical understanding of generalization in LLMs for AI researchers, though it is incremental as it builds on existing compression bound methods.
The authors tackled the problem of whether large language models (LLMs) can generalize beyond training data by providing the first non-vacuous generalization bounds for pretrained LLMs, showing that larger models have better bounds and are more compressible, with bounds accelerated by orders of magnitude on massive datasets.
Modern language models can contain billions of parameters, raising the question of whether they can generalize beyond the training data or simply parrot their training corpora. We provide the first non-vacuous generalization bounds for pretrained large language models (LLMs), indicating that language models are capable of discovering regularities that generalize to unseen data. In particular, we derive a compression bound that is valid for the unbounded log-likelihood loss using prediction smoothing, and we extend the bound to handle subsampling, accelerating bound computation by orders of magnitude on massive datasets. To achieve the extreme level of compression required for non-vacuous bounds, we devise SubLoRA, a simple low-dimensional nonlinear parameterization that leads to non-vacuous generalization bounds for models with nearly a billion parameters. Finally, we use our bounds to understand LLM generalization and find that larger models have better generalization bounds and are more compressible than smaller models.