Is Next Token Prediction Sufficient for GPT? Exploration on Code Logic Comprehension
This addresses the problem of limited code logic understanding in LLMs for developers and AI researchers, but it is incremental as it builds on existing pretraining methods.
The paper investigates whether next token prediction enables GPT models to comprehend code logic, finding that current LLMs perform poorly on a new logically equivalent code selection task, and proposes 'Next Token Prediction+' pretraining, which improves Code Llama and StarCoder by significant margins on this task and code completion.
Large language models (LLMs) has experienced exponential growth, they demonstrate remarkable performance across various tasks. Notwithstanding, contemporary research primarily centers on enhancing the size and quality of pretraining data, still utilizing the next token prediction task on autoregressive transformer model structure. The efficacy of this task in truly facilitating the model's comprehension of code logic remains questionable, we speculate that it still interprets code as mere text, while human emphasizes the underlying logical knowledge. In order to prove it, we introduce a new task, "Logically Equivalent Code Selection," which necessitates the selection of logically equivalent code from a candidate set, given a query code. Our experimental findings indicate that current LLMs underperform in this task, since they understand code by unordered bag of keywords. To ameliorate their performance, we propose an advanced pretraining task, "Next Token Prediction+". This task aims to modify the sentence embedding distribution of the LLM without sacrificing its generative capabilities. Our experimental results reveal that following this pretraining, both Code Llama and StarCoder, the prevalent code domain pretraining models, display significant improvements on our logically equivalent code selection task and the code completion task.