matous-volf/political-leaning-deberta-large

Text political leaning classifier based on DeBERTa V3 large

This model classifies text by its political leaning into three classes: left, center, right. It has been trained on news articles, social network posts and LLM-generated politological statements. The training data comes from the context of the United States, and so the left class is mostly defined by the liberal ideology and democratic party views, and the same applies for the right class being closely tied to the conservative and republican views.

The model is a part of the research done in the paper Predicting political leaning and politicalness of text using transformer models. It focuses on predicting political leaning as well as politicalness – a binary class indicating whether a text even is about politics or not. We have benchmarked the existing models for politicalness and shown that one of them – Political DEBATE – achieves an $F_{1}F\_1$ score of over 90 %. This makes it suitable for filtering non-political texts in front of a political leaning classifier like this one. We recommend doing so if the input to this model is not guaranteed to be about politics.

Our paper addresses the challenge of automatically classifying text according to political leaning and politicalness using transformer models. We compose a comprehensive overview of existing datasets and models for these tasks, finding that current approaches create siloed solutions that perform poorly on out-of-distribution texts. To address this limitation, we compile a diverse dataset by combining 12 datasets for political leaning classification and creating a new dataset for politicalness by extending 18 existing datasets with the appropriate label. Through extensive benchmarking with leave-one-in and leave-one-out methodologies, we evaluate the performance of existing models and train new ones with enhanced generalization capabilities.

Alongside the paper, we release the complete source code and results. This model is deployed in a demo web app. A second, smaller model has also been produced.

Usage

The model outputs 0 for the left, 1 for the center and 2 for the right leaning. The score of the predicted class is between $\frac{1}{3}\backslash frac\{1\}\{3\}$ and 1.

To use the model, you can either utilize the high-level Hugging Face pipeline:

from transformers import pipeline

pipe = pipeline(
    "text-classification",
    model="matous-volf/political-leaning-deberta-large",
    tokenizer="microsoft/deberta-v3-large",
)

text = "The government should raise taxes on the rich so it can give more money to the homeless."

output = pipe(text)
print(output)

Or load it directly:

from torch import argmax
from transformers import AutoModelForSequenceClassification, AutoTokenizer
from torch.nn.functional import softmax

tokenizer = AutoTokenizer.from_pretrained("microsoft/deberta-v3-large")
model = AutoModelForSequenceClassification.from_pretrained("matous-volf/political-leaning-deberta-large")

text = "The government should cut taxes because it is not using them efficiently anyway."

tokens = tokenizer(text, return_tensors="pt")
output = model(**tokens)
logits = output.logits

political_leaning = argmax(logits, dim=1).item()
probabilities = softmax(logits, dim=1)
score = probabilities[0, political_leaning].item()
print(political_leaning, score)

Evaluation

The following table displays the performance of the model on test sets (15 %) of the datasets used for training.

dataset	accuracy	$F_{1}F\_1$ score
Article bias prediction	89	89
BIGNEWSBLN	88.6	88.6
CommonCrawl news articles	88.9	88.9
Dem., rep. party platform topics	85.5	85.6
GPT-4 political bias	87	86.9
GPT-4 political ideologies	99.6	99.6
Media political stance	91.6	93.1
Political podcasts	99.8	99.8
Political tweets	82.1	82.1
Qbias	58	57.9
average	87	87.2

The following is an example of a confusion matrix, after evaluating the model on a test set from the CommonCrawl news articles dataset.

The complete results of all our measurements are available in the source code repository.

Training

This model is based on DeBERTa V3 large. All the datasets used for fine-tuning are listed in the paper, as well as a detailed description of the preprocessing, training and evaluation methodology. In summary, we have manually tweaked the hyperparameters with a setup designed for maximizing performance on unseen types of text (out-of-distribution) to increase the model's generalization abilities. In this setup, we have left one of the datasets at a time out of the training sample and used it as the validation set. Then, we have taken the resulting optimal hyperparameters and trained this model on all the available datasets.

Authors

• Matous Volf ([email protected]), DELTA – High school of computer science and economics, Pardubice, Czechia
• Jakub Simko ([email protected]), Kempelen Institute of Intelligent Technologies, Bratislava, Slovakia

Citation

BibTeX

@article{volf-simko-2025-political-leaning,
  title        = {Predicting political leaning and politicalness of text using transformer models},
  author       = {Volf, Matous and Simko, Jakub},
  year         = 2025,
  institution  = {DELTA – High school of computer science and economics, Pardubice, Czechia; Kempelen Institute of Intelligent Technologies, Bratislava, Slovakia}
}

APA

Volf, M. and Simko, J. (2025). Predicting political leaning and politicalness of text using transformer models. DELTA – High school of computer science and economics, Pardubice, Czechia; Kempelen Institute of Intelligent Technologies, Bratislava, Slovakia.