ESCOXLM-R: Multilingual Taxonomy-driven Pre-training for the Job Market
  Domain
Mike Zhang Rob van der Goot Barbara Plank
The increasing number of benchmarks for Natural Language Processing (NLP)
tasks in the computational job market domain highlights the demand for methods
that can handle job-related tasks such as skill extraction, skill
classification, job title classification, and de-identification. While some
approaches have been developed that are specific to the job market domain,
there is a lack of generalized, multilingual models and benchmarks for these
tasks. In this study, we introduce a language model called ESCOXLM-R, based on
XLM-R, which uses domain-adaptive pre-training on the European Skills,
Competences, Qualifications and Occupations (ESCO) taxonomy, covering 27
languages. The pre-training objectives for ESCOXLM-R include dynamic masked
language modeling and a novel additional objective for inducing multilingual
taxonomical ESCO relations. We comprehensively evaluate the performance of
ESCOXLM-R on 6 sequence labeling and 3 classification tasks in 4 languages and
find that it achieves state-of-the-art results on 6 out of 9 datasets. Our
analysis reveals that ESCOXLM-R performs better on short spans and outperforms
XLM-R on entity-level and surface-level span-F1, likely due to ESCO containing
short skill and occupation titles, and encoding information on the
entity-level.