PICARD: Parsing Incrementally for Constrained Auto-Regressive Decoding from Language Models
This addresses the issue of unreliable code generation from language models for database query applications, representing an incremental improvement over existing fine-tuning methods.
The paper tackles the problem of language models generating invalid code when fine-tuned for constrained formal languages like SQL, and proposes PICARD, a method that improves performance by rejecting inadmissible tokens during decoding, achieving state-of-the-art results on Spider and CoSQL text-to-SQL tasks.
Large pre-trained language models for textual data have an unconstrained output space; at each decoding step, they can produce any of 10,000s of sub-word tokens. When fine-tuned to target constrained formal languages like SQL, these models often generate invalid code, rendering it unusable. We propose PICARD (code and trained models available at https://github.com/ElementAI/picard), a method for constraining auto-regressive decoders of language models through incremental parsing. PICARD helps to find valid output sequences by rejecting inadmissible tokens at each decoding step. On the challenging Spider and CoSQL text-to-SQL translation tasks, we show that PICARD transforms fine-tuned T5 models with passable performance into state-of-the-art solutions.