Linguistic Intelligence in Large Language Models for Telecommunications
Tasnim Ahmed Nicola Piovesan Antonio De Domenico Salimur Choudhury
Large Language Models (LLMs) have emerged as a significant advancement in the
field of Natural Language Processing (NLP), demonstrating remarkable
capabilities in language generation and other language-centric tasks. Despite
their evaluation across a multitude of analytical and reasoning tasks in
various scientific domains, a comprehensive exploration of their knowledge and
understanding within the realm of natural language tasks in the
telecommunications domain is still needed. This study, therefore, seeks to
evaluate the knowledge and understanding capabilities of LLMs within this
domain. To achieve this, we conduct an exhaustive zero-shot evaluation of four
prominent LLMs-Llama-2, Falcon, Mistral, and Zephyr. These models require fewer
resources than ChatGPT, making them suitable for resource-constrained
environments. Their performance is compared with state-of-the-art, fine-tuned
models. To the best of our knowledge, this is the first work to extensively
evaluate and compare the understanding of LLMs across multiple language-centric
tasks in this domain. Our evaluation reveals that zero-shot LLMs can achieve
performance levels comparable to the current state-of-the-art fine-tuned
models. This indicates that pretraining on extensive text corpora equips LLMs
with a degree of specialization, even within the telecommunications domain. We
also observe that no single LLM consistently outperforms others, and the
performance of different LLMs can fluctuate. Although their performance lags
behind fine-tuned models, our findings underscore the potential of LLMs as a
valuable resource for understanding various aspects of this field that lack
large annotated data.