Autoencoding Pixies: Amortised Variational Inference with Graph Convolutions for Functional Distributional Semantics
This work addresses a bottleneck in training linguistically interpretable semantic models, offering an incremental improvement for computational linguistics.
The paper tackles the computational expense and slow convergence of Functional Distributional Semantics by introducing the Pixie Autoencoder, which uses a graph-convolutional neural network for amortised variational inference, achieving better results on semantic similarity in context and semantic composition tasks and outperforming BERT.
Functional Distributional Semantics provides a linguistically interpretable framework for distributional semantics, by representing the meaning of a word as a function (a binary classifier), instead of a vector. However, the large number of latent variables means that inference is computationally expensive, and training a model is therefore slow to converge. In this paper, I introduce the Pixie Autoencoder, which augments the generative model of Functional Distributional Semantics with a graph-convolutional neural network to perform amortised variational inference. This allows the model to be trained more effectively, achieving better results on two tasks (semantic similarity in context and semantic composition), and outperforming BERT, a large pre-trained language model.