Text vectorization via transformer-based language models and n-gram
  perplexities
Mihailo Škorić
As the probability (and thus perplexity) of a text is calculated based on the
product of the probabilities of individual tokens, it may happen that one
unlikely token significantly reduces the probability (i.e., increase the
perplexity) of some otherwise highly probable input, while potentially
representing a simple typographical error. Also, given that perplexity is a
scalar value that refers to the entire input, information about the probability
distribution within it is lost in the calculation (a relatively good text that
has one unlikely token and another text in which each token is equally likely
they can have the same perplexity value), especially for longer texts. As an
alternative to scalar perplexity this research proposes a simple algorithm used
to calculate vector values based on n-gram perplexities within the input. Such
representations consider the previously mentioned aspects, and instead of a
unique value, the relative perplexity of each text token is calculated, and
these values are combined into a single vector representing the input.