OMPGPT: A Generative Pre-trained Transformer Model for OpenMP
Le Chen Arijit Bhattacharjee Nesreen Ahmed Niranjan Hasabnis Gal Oren Vy Vo Ali Jannesari
Large language models (LLMs), as epitomized by models like ChatGPT, have
revolutionized the field of natural language processing (NLP). Along with this
trend, code-based large language models such as StarCoder, WizardCoder, and
CodeLlama have emerged, trained extensively on vast repositories of code data.
Yet, inherent in their design, these models primarily focus on generative tasks
like code generation, code completion, and comment generation, and general
support for multiple programming languages. While the generic abilities of code
LLMs are useful for many programmers, the area of high-performance computing
(HPC) has a narrower set of requirements that make a smaller and more
domain-specific LM a smarter choice. This paper introduces OMPGPT, a novel
model meticulously designed to harness the inherent strengths of language
models for OpenMP pragma generation. Furthermore, we adopt and adapt prompt
engineering techniques from the NLP domain to create chain-of-OMP, an
innovative strategy designed to enhance OMPGPT's effectiveness. Our extensive
evaluations demonstrate that OMPGPT outperforms existing large language models
specialized in OpenMP tasks and maintains a notably smaller size, aligning it
more closely with the typical hardware constraints of HPC environments. We
consider our contribution as a pivotal bridge, connecting the advantage of
language models with the specific demands of HPC tasks. The success of OMPGPT
lays a solid foundation, suggesting its potential applicability and
adaptability to a wider range of HPC tasks, thereby opening new avenues in the
field of computational efficiency and effectiveness.