Graph-Induced Syntactic-Semantic Spaces in Transformer-Based Variational AutoEncoders
This work addresses the limitation of existing syntactic injection methods to LSTM-based VAEs, offering an incremental improvement for natural language processing and mathematical expression generation.
The paper tackled the problem of injecting syntactic information into Transformer-based Variational AutoEncoders (VAEs) to improve latent space organization and reduce information loss, resulting in enhanced performance on language modeling and generation tasks.
The injection of syntactic information in Variational AutoEncoders (VAEs) has been shown to result in an overall improvement of performances and generalisation. An effective strategy to achieve such a goal is to separate the encoding of distributional semantic features and syntactic structures into heterogeneous latent spaces via multi-task learning or dual encoder architectures. However, existing works employing such techniques are limited to LSTM-based VAEs. In this paper, we investigate latent space separation methods for structural syntactic injection in Transformer-based VAE architectures (i.e., Optimus). Specifically, we explore how syntactic structures can be leveraged in the encoding stage through the integration of graph-based and sequential models, and how multiple, specialised latent representations can be injected into the decoder's attention mechanism via low-rank operators. Our empirical evaluation, carried out on natural language sentences and mathematical expressions, reveals that the proposed end-to-end VAE architecture can result in a better overall organisation of the latent space, alleviating the information loss occurring in standard VAE setups, resulting in enhanced performances on language modelling and downstream generation tasks.