Performance Analysis of Transformer Based Models (BERT, ALBERT and
  RoBERTa) in Fake News Detection
Shafna Fitria Nur Azizah Hasan Dwi Cahyono Sari Widya Sihwi Wisnu Widiarto
Fake news is fake material in a news media format but is not processed
properly by news agencies. The fake material can provoke or defame significant
entities or individuals or potentially even for the personal interests of the
creators, causing problems for society. Distinguishing fake news and real news
is challenging due to limited of domain knowledge and time constraints.
According to the survey, the top three areas most exposed to hoaxes and
misinformation by residents are in Banten, DKI Jakarta and West Java. The model
of transformers is referring to an approach in the field of artificial
intelligence (AI) in natural language processing utilizing the deep learning
architectures. Transformers exercise a powerful attention mechanism to process
text in parallel and produce rich and contextual word representations. A
previous study indicates a superior performance of a transformer model known as
BERT over and above non transformer approach. However, some studies suggest the
performance can be improved with the use of improved BERT models known as
ALBERT and RoBERTa. However, the modified BERT models are not well explored for
detecting fake news in Bahasa Indonesia. In this research, we explore those
transformer models and found that ALBERT outperformed other models with 87.6%
accuracy, 86.9% precision, 86.9% F1-score, and 174.5 run-time (s/epoch)
respectively. Source code available at:
https://github.com/Shafna81/fakenewsdetection.git