Is Next Token Prediction Sufficient for GPT? Exploration on Code Logic
  Comprehension
Mengnan Qi Yufan Huang Yongqiang Yao Maoquan Wang Bin Gu Neel Sundaresan
Large language models (LLMs) has experienced exponential growth, they
demonstrate remarkable performance across various tasks. Notwithstanding,
contemporary research primarily centers on enhancing the size and quality of
pretraining data, still utilizing the next token prediction task on
autoregressive transformer model structure. The efficacy of this task in truly
facilitating the model's comprehension of code logic remains questionable, we
speculate that it still interprets code as mere text, while human emphasizes
the underlying logical knowledge. In order to prove it, we introduce a new
task, "Logically Equivalent Code Selection," which necessitates the selection
of logically equivalent code from a candidate set, given a query code. Our
experimental findings indicate that current LLMs underperform in this task,
since they understand code by unordered bag of keywords. To ameliorate their
performance, we propose an advanced pretraining task, "Next Token Prediction+".
This task aims to modify the sentence embedding distribution of the LLM without
sacrificing its generative capabilities. Our experimental results reveal that
following this pretraining, both Code Llama and StarCoder, the prevalent code
domain pretraining models, display significant improvements on our logically
equivalent code selection task and the code completion task.