Evaluation of large language models using an Indian language LGBTI+
  lexicon
Aditya Joshi Shruta Rawat Alpana Dange
Large language models (LLMs) are typically evaluated on the basis of
task-based benchmarks such as MMLU. Such benchmarks do not examine responsible
behaviour of LLMs in specific contexts. This is particularly true in the LGBTI+
context where social stereotypes may result in variation in LGBTI+ terminology.
Therefore, domain-specific lexicons or dictionaries may be useful as a
representative list of words against which the LLM's behaviour needs to be
evaluated. This paper presents a methodology for evaluation of LLMs using an
LGBTI+ lexicon in Indian languages. The methodology consists of four steps:
formulating NLP tasks relevant to the expected behaviour, creating prompts that
test LLMs, using the LLMs to obtain the output and, finally, manually
evaluating the results. Our qualitative analysis shows that the three LLMs we
experiment on are unable to detect underlying hateful content. Similarly, we
observe limitations in using machine translation as means to evaluate natural
language understanding in languages other than English. The methodology
presented in this paper can be useful for LGBTI+ lexicons in other languages as
well as other domain-specific lexicons. The work done in this paper opens
avenues for responsible behaviour of LLMs, as demonstrated in the context of
prevalent social perception of the LGBTI+ community.