Investigating the Performance of Language Models for Completing Code in
  Functional Programming Languages: a Haskell Case Study
Tim van Dam Frank van der Heijden Philippe de Bekker Berend Nieuwschepen Marc Otten Maliheh Izadi
Language model-based code completion models have quickly grown in use,
helping thousands of developers write code in many different programming
languages. However, research on code completion models typically focuses on
imperative languages such as Python and JavaScript, which results in a lack of
representation for functional programming languages. Consequently, these models
often perform poorly on functional languages such as Haskell. To investigate
whether this can be alleviated, we evaluate the performance of two language
models for code, CodeGPT and UniXcoder, on the functional programming language
Haskell. We fine-tune and evaluate the models on Haskell functions sourced from
a publicly accessible Haskell dataset on HuggingFace. Additionally, we manually
evaluate the models using our novel translated HumanEval dataset. Our automatic
evaluation shows that knowledge of imperative programming languages in the
pre-training of LLMs may not transfer well to functional languages, but that
code completion on functional languages is feasible. Consequently, this shows
the need for more high-quality Haskell datasets. A manual evaluation on
HumanEval-Haskell indicates CodeGPT frequently generates empty predictions and
extra comments, while UniXcoder more often produces incomplete or incorrect
predictions. Finally, we release HumanEval-Haskell, along with the fine-tuned
models and all code required to reproduce our experiments on GitHub
(https://github.com/AISE-TUDelft/HaskellCCEval).