Analysing the Impact of Removing Infrequent Words on Topic Quality in
  LDA Models
Victor Bystrov Viktoriia Naboka-Krell Anna Staszewska-Bystrova Peter Winker
An initial procedure in text-as-data applications is text preprocessing. One
of the typical steps, which can substantially facilitate computations, consists
in removing infrequent words believed to provide limited information about the
corpus. Despite popularity of vocabulary pruning, not many guidelines on how to
implement it are available in the literature. The aim of the paper is to fill
this gap by examining the effects of removing infrequent words for the quality
of topics estimated using Latent Dirichlet Allocation. The analysis is based on
Monte Carlo experiments taking into account different criteria for infrequent
terms removal and various evaluation metrics. The results indicate that pruning
is beneficial and that the share of vocabulary which might be eliminated can be
quite considerable.