Estimating Semantic Similarity between In-Domain and Out-of-Domain
  Samples
Rhitabrat Pokharel Ameeta Agrawal
Prior work typically describes out-of-domain (OOD) or out-of-distribution
(OODist) samples as those that originate from dataset(s) or source(s) different
from the training set but for the same task. When compared to in-domain (ID)
samples, the models have been known to usually perform poorer on OOD samples,
although this observation is not consistent. Another thread of research has
focused on OOD detection, albeit mostly using supervised approaches. In this
work, we first consolidate and present a systematic analysis of multiple
definitions of OOD and OODist as discussed in prior literature. Then, we
analyze the performance of a model under ID and OOD/OODist settings in a
principled way. Finally, we seek to identify an unsupervised method for
reliably identifying OOD/OODist samples without using a trained model. The
results of our extensive evaluation using 12 datasets from 4 different tasks
suggest the promising potential of unsupervised metrics in this task.