The LSCD Benchmark: a Testbed for Diachronic Word Meaning Tasks
Dominik Schlechtweg Shafqat Mumtaz Virk Nikolay Arefyev
Lexical Semantic Change Detection (LSCD) is a complex, lemma-level task,
which is usually operationalized based on two subsequently applied usage-level
tasks: First, Word-in-Context (WiC) labels are derived for pairs of usages.
Then, these labels are represented in a graph on which Word Sense Induction
(WSI) is applied to derive sense clusters. Finally, LSCD labels are derived by
comparing sense clusters over time. This modularity is reflected in most LSCD
datasets and models. It also leads to a large heterogeneity in modeling options
and task definitions, which is exacerbated by a variety of dataset versions,
preprocessing options and evaluation metrics. This heterogeneity makes it
difficult to evaluate models under comparable conditions, to choose optimal
model combinations or to reproduce results. Hence, we provide a benchmark
repository standardizing LSCD evaluation. Through transparent implementation
results become easily reproducible and by standardization different components
can be freely combined. The repository reflects the task's modularity by
allowing model evaluation for WiC, WSI and LSCD. This allows for careful
evaluation of increasingly complex model components providing new ways of model
optimization.