Graph-Induced Syntactic-Semantic Spaces in Transformer-Based Variational
  AutoEncoders
Yingji Zhang Marco Valentino Danilo S. Carvalho Ian Pratt-Hartmann André Freitas
The injection of syntactic information in Variational AutoEncoders (VAEs) has
been shown to result in an overall improvement of performances and
generalisation. An effective strategy to achieve such a goal is to separate the
encoding of distributional semantic features and syntactic structures into
heterogeneous latent spaces via multi-task learning or dual encoder
architectures. However, existing works employing such techniques are limited to
LSTM-based VAEs. In this paper, we investigate latent space separation methods
for structural syntactic injection in Transformer-based VAE architectures
(i.e., Optimus). Specifically, we explore how syntactic structures can be
leveraged in the encoding stage through the integration of graph-based and
sequential models, and how multiple, specialised latent representations can be
injected into the decoder's attention mechanism via low-rank operators. Our
empirical evaluation, carried out on natural language sentences and
mathematical expressions, reveals that the proposed end-to-end VAE architecture
can result in a better overall organisation of the latent space, alleviating
the information loss occurring in standard VAE setups, resulting in enhanced
performances on language modelling and downstream generation tasks.