Disentangling continuous and discrete linguistic signals in
  transformer-based sentence embeddings
Vivi Nastase Paola Merlo
Sentence and word embeddings encode structural and semantic information in a
distributed manner. Part of the information encoded -- particularly lexical
information -- can be seen as continuous, whereas other -- like structural
information -- is most often discrete. We explore whether we can compress
transformer-based sentence embeddings into a representation that separates
different linguistic signals -- in particular, information relevant to
subject-verb agreement and verb alternations. We show that by compressing an
input sequence that shares a targeted phenomenon into the latent layer of a
variational autoencoder-like system, the targeted linguistic information
becomes more explicit. A latent layer with both discrete and continuous
components captures better the targeted phenomena than a latent layer with only
discrete or only continuous components. These experiments are a step towards
separating linguistic signals from distributed text embeddings and linking them
to more symbolic representations.