Probabilistic Topic Modelling with Transformer Representations
Arik Reuter Anton Thielmann Christoph Weisser Benjamin Säfken Thomas Kneib
Topic modelling was mostly dominated by Bayesian graphical models during the
last decade. With the rise of transformers in Natural Language Processing,
however, several successful models that rely on straightforward clustering
approaches in transformer-based embedding spaces have emerged and consolidated
the notion of topics as clusters of embedding vectors. We propose the
Transformer-Representation Neural Topic Model (TNTM), which combines the
benefits of topic representations in transformer-based embedding spaces and
probabilistic modelling. Therefore, this approach unifies the powerful and
versatile notion of topics based on transformer embeddings with fully
probabilistic modelling, as in models such as Latent Dirichlet Allocation
(LDA). We utilize the variational autoencoder (VAE) framework for improved
inference speed and modelling flexibility. Experimental results show that our
proposed model achieves results on par with various state-of-the-art approaches
in terms of embedding coherence while maintaining almost perfect topic
diversity. The corresponding source code is available at
https://github.com/ArikReuter/TNTM.