Statistical Depth for Ranking and Characterizing Transformer-Based Text
  Embeddings
Parker Seegmiller Sarah Masud Preum
The popularity of transformer-based text embeddings calls for better
statistical tools for measuring distributions of such embeddings. One such tool
would be a method for ranking texts within a corpus by centrality, i.e.
assigning each text a number signifying how representative that text is of the
corpus as a whole. However, an intrinsic center-outward ordering of
high-dimensional text representations is not trivial. A statistical depth is a
function for ranking k-dimensional objects by measuring centrality with respect
to some observed k-dimensional distribution. We adopt a statistical depth to
measure distributions of transformer-based text embeddings, transformer-based
text embedding (TTE) depth, and introduce the practical use of this depth for
both modeling and distributional inference in NLP pipelines. We first define
TTE depth and an associated rank sum test for determining whether two corpora
differ significantly in embedding space. We then use TTE depth for the task of
in-context learning prompt selection, showing that this approach reliably
improves performance over statistical baseline approaches across six text
classification tasks. Finally, we use TTE depth and the associated rank sum
test to characterize the distributions of synthesized and human-generated
corpora, showing that five recent synthetic data augmentation processes cause a
measurable distributional shift away from associated human-generated text.