LUNA: A Framework for Language Understanding and Naturalness Assessment
Marat Saidov Aleksandra Bakalova Ekaterina Taktasheva Vladislav Mikhailov Ekaterina Artemova
The evaluation of Natural Language Generation (NLG) models has gained
increased attention, urging the development of metrics that evaluate various
aspects of generated text. LUNA addresses this challenge by introducing a
unified interface for 20 NLG evaluation metrics. These metrics are categorized
based on their reference-dependence and the type of text representation they
employ, from string-based n-gram overlap to the utilization of static
embeddings and pre-trained language models.
  The straightforward design of LUNA allows for easy extension with novel
metrics, requiring just a few lines of code. LUNA offers a user-friendly tool
for evaluating generated texts.