Studying LLM Performance on Closed- and Open-source Data
Toufique Ahmed Christian Bird Premkumar Devanbu Saikat Chakraborty
Large Language models (LLMs) are finding wide use in software engineering
practice. These models are extremely data-hungry, and are largely trained on
open-source (OSS) code distributed with permissive licenses. In terms of actual
use however, a great deal of software development still occurs in the
for-profit/proprietary sphere, where the code under development is not, and
never has been, in the public domain; thus, many developers, do their work, and
use LLMs, in settings where the models may not be as familiar with the code
under development. In such settings, do LLMs work as well as they do for OSS
code? If not, what are the differences? When performance differs, what are the
possible causes, and are there work-arounds? In this paper, we examine this
issue using proprietary, closed-source software data from Microsoft, where most
proprietary code is in C# and C++. We find that performance for C# changes
little from OSS --> proprietary code, but does significantly reduce for C++; we
find that this difference is attributable to differences in identifiers. We
also find that some performance degradation, in some cases, can be ameliorated
efficiently by in-context learning.