Extending the Frontier of ChatGPT: Code Generation and Debugging
Fardin Ahsan Sakib Saadat Hasan Khan A. H. M. Rezaul Karim
Large-scale language models (LLMs) have emerged as a groundbreaking
innovation in the realm of question-answering and conversational agents. These
models, leveraging different deep learning architectures such as Transformers,
are trained on vast corpora to predict sentences based on given queries. Among
these LLMs, ChatGPT, developed by OpenAI, has ushered in a new era by utilizing
artificial intelligence (AI) to tackle diverse problem domains, ranging from
composing essays and biographies to solving intricate mathematical integrals.
The versatile applications enabled by ChatGPT offer immense value to users.
However, assessing the performance of ChatGPT's output poses a challenge,
particularly in scenarios where queries lack clear objective criteria for
correctness. For instance, evaluating the quality of generated essays becomes
arduous and relies heavily on manual labor, in stark contrast to evaluating
solutions to well-defined, closed-ended questions such as mathematical
problems. This research paper delves into the efficacy of ChatGPT in solving
programming problems, examining both the correctness and the efficiency of its
solution in terms of time and memory complexity. The research reveals a
commendable overall success rate of 71.875\%, denoting the proportion of
problems for which ChatGPT was able to provide correct solutions that
successfully satisfied all the test cases present in Leetcode. It exhibits
strengths in structured problems and shows a linear correlation between its
success rate and problem acceptance rates. However, it struggles to improve
solutions based on feedback, pointing to potential shortcomings in debugging
tasks. These findings provide a compact yet insightful glimpse into ChatGPT's
capabilities and areas for improvement.