Reinforcement Learning for Topic Models
Jeremy Costello Marek Z. Reformat
We apply reinforcement learning techniques to topic modeling by replacing the
variational autoencoder in ProdLDA with a continuous action space reinforcement
learning policy. We train the system with a policy gradient algorithm
REINFORCE. Additionally, we introduced several modifications: modernize the
neural network architecture, weight the ELBO loss, use contextual embeddings,
and monitor the learning process via computing topic diversity and coherence
for each training step. Experiments are performed on 11 data sets. Our
unsupervised model outperforms all other unsupervised models and performs on
par with or better than most models using supervised labeling. Our model is
outperformed on certain data sets by a model using supervised labeling and
contrastive learning. We have also conducted an ablation study to provide
empirical evidence of performance improvements from changes we made to ProdLDA
and found that the reinforcement learning formulation boosts performance.