Beware of Words: Evaluating the Lexical Richness of Conversational Large
  Language Models
Gonzalo Martínez José Alberto Hernández Javier Conde Pedro Reviriego Elena Merino
The performance of conversational Large Language Models (LLMs) in general,
and of ChatGPT in particular, is currently being evaluated on many different
tasks, from logical reasoning or maths to answering questions on a myriad of
topics. Instead, much less attention is being devoted to the study of the
linguistic features of the texts generated by these LLMs. This is surprising
since LLMs are models for language, and understanding how they use the language
is important. Indeed, conversational LLMs are poised to have a significant
impact on the evolution of languages as they may eventually dominate the
creation of new text. This means that for example, if conversational LLMs do
not use a word it may become less and less frequent and eventually stop being
used altogether. Therefore, evaluating the linguistic features of the text they
produce and how those depend on the model parameters is the first step toward
understanding the potential impact of conversational LLMs on the evolution of
languages. In this paper, we consider the evaluation of the lexical richness of
the text generated by LLMs and how it depends on the model parameters. A
methodology is presented and used to conduct a comprehensive evaluation of
lexical richness using ChatGPT as a case study. The results show how lexical
richness depends on the version of ChatGPT and some of its parameters, such as
the presence penalty, or on the role assigned to the model. The dataset and
tools used in our analysis are released under open licenses with the goal of
drawing the much-needed attention to the evaluation of the linguistic features
of LLM-generated text.