Quasi-symbolic Semantic Geometry over Transformer-based Variational AutoEncoder
This addresses the need for better controllability and interpretability in language models, though it appears incremental as it builds on existing Transformer and VAE methods.
The authors tackled the problem of making sentence representations from language models more controllable and interpretable by proposing a formal semantic geometry framework based on semantic role-word content composition. They implemented this using a Transformer-based Variational AutoEncoder with GPT2, showing it can potentially improve control and interpretation of sentence generation.
Formal/symbolic semantics can provide canonical, rigid controllability and interpretability to sentence representations due to their \textit{localisation} or \textit{composition} property. How can we deliver such property to the current distributional sentence representations to control and interpret the generation of language models (LMs)? In this work, we theoretically frame the sentence semantics as the composition of \textit{semantic role - word content} features and propose the formal semantic geometry. To inject such geometry into Transformer-based LMs (i.e. GPT2), we deploy Transformer-based Variational AutoEncoder with a supervision approach, where the sentence generation can be manipulated and explained over low-dimensional latent Gaussian space. In addition, we propose a new probing algorithm to guide the movement of sentence vectors over such geometry. Experimental results reveal that the formal semantic geometry can potentially deliver better control and interpretation to sentence generation.