Utilizing Social Media Attributes for Enhanced Keyword Detection: An
  IDF-LDA Model Applied to Sina Weibo
Yifei Yue
With the rapid development of social media such as Twitter and Weibo,
detecting keywords from a huge volume of text data streams in real-time has
become a critical problem. The keyword detection problem aims at searching
important information from massive text data to reflect the most important
events or topics. However, social media data usually has unique features: the
documents are usually short, the language is colloquial, and the data is likely
to have significant temporal patterns. Therefore, it could be challenging to
discover critical information from these text streams. In this paper, we
propose a novel method to address the keyword detection problem in social
media. Our model combines the Inverse Document Frequency (IDF) and Latent
Dirichlet Allocation (LDA) models to better cope with the distinct attributes
of social media data, such as the number of likes, comments, and retweets. By
weighting the importance of each document based on these attributes, our method
can effectively detect more representative keywords over time. Comprehensive
experiments conducted under various conditions on Weibo data illustrate that
our approach outperforms the baselines in various evaluation metrics, including
precision and recall for multiple problem settings.