An Exploration of Encoder-Decoder Approaches to Multi-Label
  Classification for Legal and Biomedical Text
Yova Kementchedjhieva Ilias Chalkidis
Standard methods for multi-label text classification largely rely on
encoder-only pre-trained language models, whereas encoder-decoder models have
proven more effective in other classification tasks. In this study, we compare
four methods for multi-label classification, two based on an encoder only, and
two based on an encoder-decoder. We carry out experiments on four datasets --
two in the legal domain and two in the biomedical domain, each with two levels
of label granularity -- and always depart from the same pre-trained model, T5.
Our results show that encoder-decoder methods outperform encoder-only methods,
with a growing advantage on more complex datasets and labeling schemes of finer
granularity. Using encoder-decoder models in a non-autoregressive fashion, in
particular, yields the best performance overall, so we further study this
approach through ablations to better understand its strengths.