SMILE: Evaluation and Domain Adaptation for Social Media Language
  Understanding
Vasilisa Bashlovkina Riley Matthews Zhaobin Kuang Simon Baumgartner Michael Bendersky
We study the ability of transformer-based language models (LMs) to understand
social media language. Social media (SM) language is distinct from standard
written language, yet existing benchmarks fall short of capturing LM
performance in this socially, economically, and politically important domain.
We quantify the degree to which social media language differs from conventional
language and conclude that the difference is significant both in terms of token
distribution and rate of linguistic shift. Next, we introduce a new benchmark
for Social MedIa Language Evaluation (SMILE) that covers four SM platforms and
eleven tasks. Finally, we show that learning a tokenizer and pretraining on a
mix of social media and conventional language yields an LM that outperforms the
best similar-sized alternative by 4.2 points on the overall SMILE score.