MCFEND: A Multi-source Benchmark Dataset for Chinese Fake News Detection
Yupeng Li Haorui He Jin Bai Dacheng Wen
The prevalence of fake news across various online sources has had a
significant influence on the public. Existing Chinese fake news detection
datasets are limited to news sourced solely from Weibo. However, fake news
originating from multiple sources exhibits diversity in various aspects,
including its content and social context. Methods trained on purely one single
news source can hardly be applicable to real-world scenarios. Our pilot
experiment demonstrates that the F1 score of the state-of-the-art method that
learns from a large Chinese fake news detection dataset, Weibo-21, drops
significantly from 0.943 to 0.470 when the test data is changed to multi-source
news data, failing to identify more than one-third of the multi-source fake
news. To address this limitation, we constructed the first multi-source
benchmark dataset for Chinese fake news detection, termed MCFEND, which is
composed of news we collected from diverse sources such as social platforms,
messaging apps, and traditional online news outlets. Notably, such news has
been fact-checked by 14 authoritative fact-checking agencies worldwide. In
addition, various existing Chinese fake news detection methods are thoroughly
evaluated on our proposed dataset in cross-source, multi-source, and unseen
source ways. MCFEND, as a benchmark dataset, aims to advance Chinese fake news
detection approaches in real-world scenarios.