Natural Language Generation and Understanding of Big Code for
  AI-Assisted Programming: A Review
Man Fai Wong Shangxin Guo Ching Nam Hang Siu Wai Ho Chee Wei Tan
This paper provides a comprehensive review of the literature concerning the
utilization of Natural Language Processing (NLP) techniques, with a particular
focus on transformer-based large language models (LLMs) trained using Big Code,
within the domain of AI-assisted programming tasks. LLMs, augmented with
software naturalness, have played a crucial role in facilitating AI-assisted
programming applications, including code generation, code completion, code
translation, code refinement, code summarization, defect detection, and clone
detection. Notable examples of such applications include the GitHub Copilot
powered by OpenAI's Codex and DeepMind AlphaCode. This paper presents an
overview of the major LLMs and their applications in downstream tasks related
to AI-assisted programming. Furthermore, it explores the challenges and
opportunities associated with incorporating NLP techniques with software
naturalness in these applications, with a discussion on extending AI-assisted
programming capabilities to Apple's Xcode for mobile software development. This
paper also presents the challenges of and opportunities for incorporating NLP
techniques with software naturalness, empowering developers with advanced
coding assistance and streamlining the software development process.