A Side-by-side Comparison of Transformers for English Implicit Discourse
  Relation Classification
Bruce W. Lee BongSeok Yang Jason Hyung-Jong Lee
Though discourse parsing can help multiple NLP fields, there has been no wide
language model search done on implicit discourse relation classification. This
hinders researchers from fully utilizing public-available models in discourse
analysis. This work is a straightforward, fine-tuned discourse performance
comparison of seven pre-trained language models. We use PDTB-3, a popular
discourse relation annotated dataset. Through our model search, we raise SOTA
to 0.671 ACC and obtain novel observations. Some are contrary to what has been
reported before (Shi and Demberg, 2019b), that sentence-level pre-training
objectives (NSP, SBO, SOP) generally fail to produce the best performing model
for implicit discourse relation classification. Counterintuitively,
similar-sized PLMs with MLM and full attention led to better performance.