Learning with Latent Structures in Natural Language Processing: A Survey
It addresses the challenge of integrating non-differentiable latent structures into gradient-based optimization for NLP researchers and practitioners, but is incremental as it reviews existing methods.
This survey examines methods for learning with latent discrete structures in NLP to enhance performance and interpretability, covering three main families of approaches and their applications.
While end-to-end learning with fully differentiable models has enabled tremendous success in natural language process (NLP) and machine learning, there have been significant recent interests in learning with latent discrete structures to incorporate better inductive biases for improved end-task performance and better interpretability. This paradigm, however, is not straightforwardly amenable to the mainstream gradient-based optimization methods. This work surveys three main families of methods to learn such models: surrogate gradients, continuous relaxation, and marginal likelihood maximization via sampling. We conclude with a review of applications of these methods and an inspection of the learned latent structure that they induce.