Understanding writing style in social media with a supervised
  contrastively pre-trained transformer
Javier Huertas-Tato Alejandro Martin David Camacho
Online Social Networks serve as fertile ground for harmful behavior, ranging
from hate speech to the dissemination of disinformation. Malicious actors now
have unprecedented freedom to misbehave, leading to severe societal unrest and
dire consequences, as exemplified by events such as the Capitol assault during
the US presidential election and the Antivaxx movement during the COVID-19
pandemic. Understanding online language has become more pressing than ever.
While existing works predominantly focus on content analysis, we aim to shift
the focus towards understanding harmful behaviors by relating content to their
respective authors. Numerous novel approaches attempt to learn the stylistic
features of authors in texts, but many of these approaches are constrained by
small datasets or sub-optimal training losses. To overcome these limitations,
we introduce the Style Transformer for Authorship Representations (STAR),
trained on a large corpus derived from public sources of 4.5 x 10^6 authored
texts involving 70k heterogeneous authors. Our model leverages Supervised
Contrastive Loss to teach the model to minimize the distance between texts
authored by the same individual. This author pretext pre-training task yields
competitive performance at zero-shot with PAN challenges on attribution and
clustering. Additionally, we attain promising results on PAN verification
challenges using a single dense layer, with our model serving as an embedding
encoder. Finally, we present results from our test partition on Reddit. Using a
support base of 8 documents of 512 tokens, we can discern authors from sets of
up to 1616 authors with at least 80\% accuracy. We share our pre-trained model
at huggingface (https://huggingface.co/AIDA-UPM/star) and our code is available
at (https://github.com/jahuerta92/star)