Comparing effectiveness of regularization methods on text
  classification: Simple and complex model in data shortage situation
Jongga Lee Jaeseung Yim Seohee Park Changwon Lim
Text classification is the task of assigning a document to a predefined
class. However, it is expensive to acquire enough labeled documents or to label
them. In this paper, we study the regularization methods' effects on various
classification models when only a few labeled data are available. We compare a
simple word embedding-based model, which is simple but effective, with complex
models (CNN and BiLSTM). In supervised learning, adversarial training can
further regularize the model. When an unlabeled dataset is available, we can
regularize the model using semi-supervised learning methods such as the Pi
model and virtual adversarial training. We evaluate the regularization effects
on four text classification datasets (AG news, DBpedia, Yahoo! Answers, Yelp
Polarity), using only 0.1% to 0.5% of the original labeled training documents.
The simple model performs relatively well in fully supervised learning, but
with the help of adversarial training and semi-supervised learning, both simple
and complex models can be regularized, showing better results for complex
models. Although the simple model is robust to overfitting, a complex model
with well-designed prior beliefs can be also robust to overfitting.