Automatic detection of relevant information, predictions and forecasts
  in financial news through topic modelling with Latent Dirichlet Allocation
Silvia García-Méndez Francisco de Arriba-Pérez Ana Barros-Vila Francisco J. González-Castaño Enrique Costa-Montenegro
Financial news items are unstructured sources of information that can be
mined to extract knowledge for market screening applications. Manual extraction
of relevant information from the continuous stream of finance-related news is
cumbersome and beyond the skills of many investors, who, at most, can follow a
few sources and authors. Accordingly, we focus on the analysis of financial
news to identify relevant text and, within that text, forecasts and
predictions. We propose a novel Natural Language Processing (NLP) system to
assist investors in the detection of relevant financial events in unstructured
textual sources by considering both relevance and temporality at the discursive
level. Firstly, we segment the text to group together closely related text.
Secondly, we apply co-reference resolution to discover internal dependencies
within segments. Finally, we perform relevant topic modelling with Latent
Dirichlet Allocation (LDA) to separate relevant from less relevant text and
then analyse the relevant text using a Machine Learning-oriented temporal
approach to identify predictions and speculative statements. We created an
experimental data set composed of 2,158 financial news items that were manually
labelled by NLP researchers to evaluate our solution. The ROUGE-L values for
the identification of relevant text and predictions/forecasts were 0.662 and
0.982, respectively. To our knowledge, this is the first work to jointly
consider relevance and temporality at the discursive level. It contributes to
the transfer of human associative discourse capabilities to expert systems
through the combination of multi-paragraph topic segmentation and co-reference
resolution to separate author expression patterns, topic modelling with LDA to
detect relevant text, and discursive temporality analysis to identify forecasts
and predictions within this text.