Exploring the Impact of Corpus Diversity on Financial Pretrained
  Language Models
Jaeyoung Choe Keonwoong Noh Nayeon Kim Seyun Ahn Woohwan Jung
Over the past few years, various domain-specific pretrained language models
(PLMs) have been proposed and have outperformed general-domain PLMs in
specialized areas such as biomedical, scientific, and clinical domains. In
addition, financial PLMs have been studied because of the high economic impact
of financial data analysis. However, we found that financial PLMs were not
pretrained on sufficiently diverse financial data. This lack of diverse
training data leads to a subpar generalization performance, resulting in
general-purpose PLMs, including BERT, often outperforming financial PLMs on
many downstream tasks. To address this issue, we collected a broad range of
financial corpus and trained the Financial Language Model (FiLM) on these
diverse datasets. Our experimental results confirm that FiLM outperforms not
only existing financial PLMs but also general domain PLMs. Furthermore, we
provide empirical evidence that this improvement can be achieved even for
unseen corpus groups.