LLMs are Not Just Next Token Predictors
This is an incremental conceptual critique for AI researchers and theorists.
The paper argues that viewing LLMs solely as next token predictors is overly reductive and misses important explanations of their behavior and capabilities, drawing an analogy to the gene's eye view in biology.
LLMs are statistical models of language learning through stochastic gradient descent with a next token prediction objective. Prompting a popular view among AI modelers: LLMs are just next token predictors. While LLMs are engineered using next token prediction, and trained based on their success at this task, our view is that a reduction to just next token predictor sells LLMs short. Moreover, there are important explanations of LLM behavior and capabilities that are lost when we engage in this kind of reduction. In order to draw this out, we will make an analogy with a once prominent research program in biology explaining evolution and development from the gene's eye view.