Domain-Specific Code Language Models: Unraveling the Potential for HPC
  Codes and Tasks
Tal Kadosh Niranjan Hasabnis Vy A. Vo Nadav Schneider Neva Krien Mihai Capota Abdul Wasay Nesreen Ahmed Ted Willke Guy Tamir Yuval Pinter Timothy Mattson Gal Oren
With easier access to powerful compute resources, there is a growing trend in
AI for software development to develop larger language models (LLMs) to address
a variety of programming tasks. Even LLMs applied to tasks from the
high-performance computing (HPC) domain are huge in size and demand expensive
compute resources for training. This is partly because these LLMs for HPC tasks
are obtained by finetuning existing LLMs that support several natural and/or
programming languages. We found this design choice confusing - why do we need
large LMs trained on natural languages and programming languages unrelated to
HPC for HPC-specific tasks?
  In this line of work, we aim to question choices made by existing LLMs by
developing smaller LMs for specific domains - we call them domain-specific LMs.
Specifically, we start off with HPC as a domain and build an HPC-specific LM,
named MonoCoder, that is orders of magnitude smaller than existing LMs but
delivers similar, if not better performance, on non-HPC and HPC tasks.
Specifically, we pre-trained MonoCoder on an HPC-specific dataset (named
HPCorpus) of C and C++ programs mined from GitHub. We evaluated the performance
of MonoCoder against conventional multi-lingual LLMs. Results demonstrate that
MonoCoder, although much smaller than existing LMs, achieves similar results on
normalized-perplexity tests and much better ones in CodeBLEU competence for
high-performance and parallel code generations. Furthermore, fine-tuning the
base model for the specific task of parallel code generation (OpenMP parallel
for pragmas) demonstrates outstanding results compared to GPT, especially when
local misleading semantics are removed by our novel pre-processor Tokompiler,
showcasing the ability of domain-specific models to assist in HPC-relevant
tasks.