Language Generation via Combinatorial Constraint Satisfaction: A Tree Search Enhanced Monte-Carlo Approach
This work provides a flexible, training-free method for controllable text generation, which is important for researchers and practitioners needing to generate text with specific structural or semantic properties.
This paper addresses the problem of generating natural language under complex combinatorial constraints. The proposed TSMH method, which integrates a tree search algorithm into a Monte Carlo approach, generates high-likelihood sentences that satisfy these constraints, achieving consistent and significant improvements across multiple language generation tasks.
Generating natural language under complex constraints is a principled formulation towards controllable text generation. We present a framework to allow specification of combinatorial constraints for sentence generation. We propose TSMH, an efficient method to generate high likelihood sentences with respect to a pre-trained language model while satisfying the constraints. Our approach is highly flexible, requires no task-specific training, and leverages efficient constraint satisfaction solving techniques. To better handle the combinatorial constraints, a tree search algorithm is embedded into the proposal process of the Markov chain Monte Carlo (MCMC) to explore candidates that satisfy more constraints. Compared to existing MCMC approaches, our sampling approach has a better mixing performance. Experiments show that TSMH achieves consistent and significant improvement on multiple language generation tasks.