EcomGPT-CT: Continual Pre-training of E-commerce Large Language Models
  with Semi-structured Data
Shirong Ma Shen Huang Shulin Huang Xiaobin Wang Yangning Li Hai-Tao Zheng Pengjun Xie Fei Huang Yong Jiang
Large Language Models (LLMs) pre-trained on massive corpora have exhibited
remarkable performance on various NLP tasks. However, applying these models to
specific domains still poses significant challenges, such as lack of domain
knowledge, limited capacity to leverage domain knowledge and inadequate
adaptation to domain-specific data formats. Considering the exorbitant cost of
training LLMs from scratch and the scarcity of annotated data within particular
domains, in this work, we focus on domain-specific continual pre-training of
LLMs using E-commerce domain as an exemplar. Specifically, we explore the
impact of continual pre-training on LLMs employing unlabeled general and
E-commercial corpora. Furthermore, we design a mixing strategy among different
data sources to better leverage E-commercial semi-structured data. We construct
multiple tasks to assess LLMs' few-shot In-context Learning ability and their
zero-shot performance after instruction tuning in E-commerce domain.
Experimental results demonstrate the effectiveness of continual pre-training of
E-commerce LLMs and the efficacy of our devised data mixing strategy.