XuanYuan 2.0: A Large Chinese Financial Chat Model with Hundreds of
  Billions Parameters
Xuanyu Zhang Qing Yang Dongliang Xu
In recent years, pre-trained language models have undergone rapid development
with the emergence of large-scale models. However, there is a lack of
open-sourced chat models specifically designed for the Chinese language,
especially in the field of Chinese finance, at the scale of hundreds of
billions. To address this gap, we introduce XuanYuan 2.0, the largest Chinese
chat model to date, built upon the BLOOM-176B architecture. Additionally, we
propose a novel training method called hybrid-tuning to mitigate catastrophic
forgetting. By combining general-domain with domain-specific knowledge and
integrating the stages of pre-training and fine-tuning, XuanYuan 2.0 is capable
of providing accurate and contextually appropriate responses in the Chinese
financial domain.