Understanding Telecom Language Through Large Language Models
Lina Bariah Hang Zou Qiyang Zhao Belkacem Mouhouche Faouzi Bader Merouane Debbah
The recent progress of artificial intelligence (AI) opens up new frontiers in
the possibility of automating many tasks involved in Telecom networks design,
implementation, and deployment. This has been further pushed forward with the
evolution of generative artificial intelligence (AI), including the emergence
of large language models (LLMs), which is believed to be the cornerstone toward
realizing self-governed, interactive AI agents. Motivated by this, in this
paper, we aim to adapt the paradigm of LLMs to the Telecom domain. In
particular, we fine-tune several LLMs including BERT, distilled BERT, RoBERTa
and GPT-2, to the Telecom domain languages, and demonstrate a use case for
identifying the 3rd Generation Partnership Project (3GPP) standard working
groups. We consider training the selected models on 3GPP technical documents
(Tdoc) pertinent to years 2009-2019 and predict the Tdoc categories in years
2020-2023. The results demonstrate that fine-tuning BERT and RoBERTa model
achieves 84.6% accuracy, while GPT-2 model achieves 83% in identifying 3GPP
working groups. The distilled BERT model with around 50% less parameters
achieves similar performance as others. This corroborates that fine-tuning
pretrained LLM can effectively identify the categories of Telecom language. The
developed framework shows a stepping stone towards realizing intent-driven and
self-evolving wireless networks from Telecom languages, and paves the way for
the implementation of generative AI in the Telecom domain.