Combining Constrained and Unconstrained Decoding via Boosting: BoostCD and Its Application to Information Extraction
This addresses a key bottleneck in structured NLP tasks like information extraction, offering a method to enhance output quality without retraining, though it is incremental as it builds on existing constrained decoding approaches.
The paper tackles the problem of low-quality output in structured NLP tasks when using constrained decoding with autoregressive language models, by introducing Boosted Constrained Decoding (BoostCD) that combines constrained and unconstrained decoding phases to improve performance, as demonstrated with BoostIE achieving superior results in closed information extraction both in and out of distribution.
Many recent approaches to structured NLP tasks use an autoregressive language model $M$ to map unstructured input text $x$ to output text $y$ representing structured objects (such as tuples, lists, trees, code, etc.), where the desired output structure is enforced via constrained decoding. During training, these approaches do not require the model to be aware of the constraints, which are merely implicit in the training outputs $y$. This is advantageous as it allows for dynamic constraints without requiring retraining, but can lead to low-quality output during constrained decoding at test time. We overcome this problem with Boosted Constrained Decoding (BoostCD), which combines constrained and unconstrained decoding in two phases: Phase 1 decodes from the base model $M$ twice, in constrained and unconstrained mode, obtaining two weak predictions. In phase 2, a learned autoregressive boosted model combines the two weak predictions into one final prediction. The mistakes made by the base model with vs. without constraints tend to be complementary, which the boosted model learns to exploit for improved performance. We demonstrate the power of BoostCD by applying it to closed information extraction. Our model, BoostIE, outperforms prior approaches both in and out of distribution, addressing several common errors identified in those approaches.