An Exploratory Literature Study on Sharing and Energy Use of Language
  Models for Source Code
Max Hort Anastasiia Grishina Leon Moonen
Large language models trained on source code can support a variety of
software development tasks, such as code recommendation and program repair.
Large amounts of data for training such models benefit the models' performance.
However, the size of the data and models results in long training times and
high energy consumption. While publishing source code allows for replicability,
users need to repeat the expensive training process if models are not shared.
The main goal of the study is to investigate if publications that trained
language models for software engineering (SE) tasks share source code and
trained artifacts. The second goal is to analyze the transparency on training
energy usage. We perform a snowballing-based literature search to find
publications on language models for source code, and analyze their reusability
from a sustainability standpoint.
  From 494 unique publications, we identified 293 relevant publications that
use language models to address code-related tasks. Among them, 27% (79 out of
293) make artifacts available for reuse. This can be in the form of tools or
IDE plugins designed for specific tasks or task-agnostic models that can be
fine-tuned for a variety of downstream tasks. Moreover, we collect insights on
the hardware used for model training, as well as training time, which together
determine the energy consumption of the development process. We find that there
are deficiencies in the sharing of information and artifacts for current
studies on source code models for software engineering tasks, with 40% of the
surveyed papers not sharing source code or trained artifacts. We recommend the
sharing of source code as well as trained artifacts, to enable sustainable
reproducibility. Moreover, comprehensive information on training times and
hardware configurations should be shared for transparency on a model's carbon
footprint.