Neural Networks Against (and For) Self-Training: Classification with
  Small Labeled and Large Unlabeled Sets
Payam Karisani
We propose a semi-supervised text classifier based on self-training using one
positive and one negative property of neural networks. One of the weaknesses of
self-training is the semantic drift problem, where noisy pseudo-labels
accumulate over iterations and consequently the error rate soars. In order to
tackle this challenge, we reshape the role of pseudo-labels and create a
hierarchical order of information. In addition, a crucial step in self-training
is to use the classifier confidence prediction to select the best candidate
pseudo-labels. This step cannot be efficiently done by neural networks, because
it is known that their output is poorly calibrated. To overcome this challenge,
we propose a hybrid metric to replace the plain confidence measurement. Our
metric takes into account the prediction uncertainty via a subsampling
technique. We evaluate our model in a set of five standard benchmarks, and show
that it significantly outperforms a set of ten diverse baseline models.
Furthermore, we show that the improvement achieved by our model is additive to
language model pretraining, which is a widely used technique for using
unlabeled documents. Our code is available at
https://github.com/p-karisani/RST.