Next Concept Prediction in Discrete Latent Space Leads to Stronger Language Models
This addresses the challenge of enhancing language model capabilities for AI applications by proposing an incremental improvement to pretraining paradigms.
The paper tackles the problem of improving language model pretraining by introducing Next Concept Prediction (NCP), which predicts discrete concepts spanning multiple tokens as a more challenging objective, resulting in consistent performance gains over traditional token-level models across 13 benchmarks and further improvements in continual pretraining on an 8B-parameter model.
We propose Next Concept Prediction (NCP), a generative pretraining paradigm built on top of Next Token Prediction (NTP). NCP predicts discrete concepts that span multiple tokens, thereby forming a more challenging pretraining objective. Our model, ConceptLM, quantizes hidden states using Vector Quantization and constructs a concept vocabulary. It leverages both NCP and NTP to drive parameter updates and generates a concept to guide the generation of the following tokens. We train ConceptLM from scratch at scales ranging from 70M to 1.5B parameters with up to 300B training data, including Pythia and GPT-2 backbones. Results on 13 benchmarks show that NCP yields consistent performance gains over traditional token-level models. Furthermore, continual pretraining experiments on an 8B-parameter Llama model indicate that NCP can further improve an NTP-trained model. Our analysis suggests that NCP leads to more powerful language models by introducing a harder pretraining task, providing a promising path toward better language modeling.