How many words does ChatGPT know? The answer is ChatWords
Gonzalo Martínez Javier Conde Pedro Reviriego Elena Merino-Gómez José Alberto Hernández Fabrizio Lombardi
The introduction of ChatGPT has put Artificial Intelligence (AI) Natural
Language Processing (NLP) in the spotlight. ChatGPT adoption has been
exponential with millions of users experimenting with it in a myriad of tasks
and application domains with impressive results. However, ChatGPT has
limitations and suffers hallucinations, for example producing answers that look
plausible but they are completely wrong. Evaluating the performance of ChatGPT
and similar AI tools is a complex issue that is being explored from different
perspectives. In this work, we contribute to those efforts with ChatWords, an
automated test system, to evaluate ChatGPT knowledge of an arbitrary set of
words. ChatWords is designed to be extensible, easy to use, and adaptable to
evaluate also other NLP AI tools. ChatWords is publicly available and its main
goal is to facilitate research on the lexical knowledge of AI tools. The
benefits of ChatWords are illustrated with two case studies: evaluating the
knowledge that ChatGPT has of the Spanish lexicon (taken from the official
dictionary of the "Real Academia Espa\~nola") and of the words that appear in
the Quixote, the well-known novel written by Miguel de Cervantes. The results
show that ChatGPT is only able to recognize approximately 80% of the words in
the dictionary and 90% of the words in the Quixote, in some cases with an
incorrect meaning. The implications of the lexical knowledge of NLP AI tools
and potential applications of ChatWords are also discussed providing directions
for further work on the study of the lexical knowledge of AI tools.