Automated stance detection in complex topics and small languages: the
  challenging case of immigration in polarizing news media
Mark Mets Andres Karjus Indrek Ibrus Maximilian Schich
Automated stance detection and related machine learning methods can provide
useful insights for media monitoring and academic research. Many of these
approaches require annotated training datasets, which limits their
applicability for languages where these may not be readily available. This
paper explores the applicability of large language models for automated stance
detection in a challenging scenario, involving a morphologically complex,
lower-resource language, and a socio-culturally complex topic, immigration. If
the approach works in this case, it can be expected to perform as well or
better in less demanding scenarios. We annotate a large set of pro and
anti-immigration examples, and compare the performance of multiple language
models as supervised learners. We also probe the usability of ChatGPT as an
instructable zero-shot classifier for the same task. Supervised achieves
acceptable performance, and ChatGPT yields similar accuracy. This is promising
as a potentially simpler and cheaper alternative for text classification tasks,
including in lower-resource languages. We further use the best-performing model
to investigate diachronic trends over seven years in two corpora of Estonian
mainstream and right-wing populist news sources, demonstrating the
applicability of the approach for news analytics and media monitoring settings,
and discuss correspondences between stance changes and real-world events.