IDoFew: Intermediate Training Using Dual-Clustering in Language Models
  for Few Labels Text Classification
Abdullah Alsuhaibani Hamad Zogan Imran Razzak Shoaib Jameel Guandong Xu
Language models such as Bidirectional Encoder Representations from
Transformers (BERT) have been very effective in various Natural Language
Processing (NLP) and text mining tasks including text classification. However,
some tasks still pose challenges for these models, including text
classification with limited labels. This can result in a cold-start problem.
Although some approaches have attempted to address this problem through
single-stage clustering as an intermediate training step coupled with a
pre-trained language model, which generates pseudo-labels to improve
classification, these methods are often error-prone due to the limitations of
the clustering algorithms. To overcome this, we have developed a novel
two-stage intermediate clustering with subsequent fine-tuning that models the
pseudo-labels reliably, resulting in reduced prediction errors. The key novelty
in our model, IDoFew, is that the two-stage clustering coupled with two
different clustering algorithms helps exploit the advantages of the
complementary algorithms that reduce the errors in generating reliable
pseudo-labels for fine-tuning. Our approach has shown significant improvements
compared to strong comparative models.