Contextualizing the Limits of Model & Evaluation Dataset Curation on
  Semantic Similarity Classification Tasks
Daniel Theron
This paper demonstrates how the limitations of pre-trained models and open
evaluation datasets factor into assessing the performance of binary semantic
similarity classification tasks. As (1) end-user-facing documentation around
the curation of these datasets and pre-trained model training regimes is often
not easily accessible and (2) given the lower friction and higher demand to
quickly deploy such systems in real-world contexts, our study reinforces prior
work showing performance disparities across datasets, embedding techniques and
distance metrics, while highlighting the importance of understanding how data
is collected, curated and analyzed in semantic similarity classification.