SentiGOLD: A Large Bangla Gold Standard Multi-Domain Sentiment Analysis
  Dataset and its Evaluation
Md. Ekramul Islam Labib Chowdhury Faisal Ahamed Khan Shazzad Hossain Sourave Hossain Mohammad Mamun Or Rashid Nabeel Mohammed Mohammad Ruhul Amin
This study introduces SentiGOLD, a Bangla multi-domain sentiment analysis
dataset. Comprising 70,000 samples, it was created from diverse sources and
annotated by a gender-balanced team of linguists. SentiGOLD adheres to
established linguistic conventions agreed upon by the Government of Bangladesh
and a Bangla linguistics committee. Unlike English and other languages, Bangla
lacks standard sentiment analysis datasets due to the absence of a national
linguistics framework. The dataset incorporates data from online video
comments, social media posts, blogs, news, and other sources while maintaining
domain and class distribution rigorously. It spans 30 domains (e.g., politics,
entertainment, sports) and includes 5 sentiment classes (strongly negative,
weakly negative, neutral, and strongly positive). The annotation scheme,
approved by the national linguistics committee, ensures a robust Inter
Annotator Agreement (IAA) with a Fleiss' kappa score of 0.88. Intra- and
cross-dataset evaluation protocols are applied to establish a standard
classification system. Cross-dataset evaluation on the noisy SentNoB dataset
presents a challenging test scenario. Additionally, zero-shot experiments
demonstrate the generalizability of SentiGOLD. The top model achieves a macro
f1 score of 0.62 (intra-dataset) across 5 classes, setting a benchmark, and
0.61 (cross-dataset from SentNoB) across 3 classes, comparable to the
state-of-the-art. Fine-tuned sentiment analysis model can be accessed at
https://sentiment.bangla.gov.bd.