Know Your Limits: Entropy Estimation Modeling for Compression and Generalization
This work addresses the challenge of improving generalization in language models for compression and prediction tasks, though it appears incremental as it builds on existing causal models with architectural modifications.
The paper tackles the problem of language model generalization by using entropy estimation to set training limits, showing that models trained to approach estimated per-token entropies achieve higher compression and better generalization than those trained without such constraints.
Language prediction is constrained by informational entropy intrinsic to language, such that there exists a limit to how accurate any language model can become and equivalently a lower bound to language compression. The most efficient language compression algorithms today are causal (next token prediction) large language models, but the use of these models to form accurate estimates of language entropy is currently computationally infeasible. We introduce encoder-augmented causal decoder model architectures that exhibit superior training efficiency characteristics and achieve higher compression than causal transformers even when trained on modest hardware. We demonstrate how entropy estimates can be obtained on a per-token basis, and show that the generalization of models trained to approach the entropy of their training data necessarily exceeds the generalization of models trained to minimize loss beyond this value. We show empirically that causal models trained to approach but not exceed estimated per-token entropies exhibit greater generalization than models trained without taking entropy into account.