Sociolinguistically Informed Interpretability: A Case Study on Hinglish
  Emotion Classification
Kushal Tatariya Heather Lent Johannes Bjerva Miryam de Lhoneux
Emotion classification is a challenging task in NLP due to the inherent
idiosyncratic and subjective nature of linguistic expression, especially with
code-mixed data. Pre-trained language models (PLMs) have achieved high
performance for many tasks and languages, but it remains to be seen whether
these models learn and are robust to the differences in emotional expression
across languages. Sociolinguistic studies have shown that Hinglish speakers
switch to Hindi when expressing negative emotions and to English when
expressing positive emotions. To understand if language models can learn these
associations, we study the effect of language on emotion prediction across 3
PLMs on a Hinglish emotion classification dataset. Using LIME and token level
language ID, we find that models do learn these associations between language
choice and emotional expression. Moreover, having code-mixed data present in
the pre-training can augment that learning when task-specific data is scarce.
We also conclude from the misclassifications that the models may overgeneralise
this heuristic to other infrequent examples where this sociolinguistic
phenomenon does not apply.