Do not be greedy, Think Twice: Sampling and Selection for Document-level Information Extraction
This addresses the challenge of output variability in DocIE for NLP practitioners, offering a more effective approach than greedy decoding, though it is incremental as it builds on existing sampling and selection techniques.
The paper tackles the problem of document-level information extraction (DocIE) by showing that sampling multiple candidate templates with an LLM and selecting the best one outperforms standard greedy decoding, with experiments demonstrating consistent improvements over baselines and state-of-the-art methods.
Document-level Information Extraction (DocIE) aims to produce an output template with the entities and relations of interest occurring in the given document. Standard practices include prompting decoder-only LLMs using greedy decoding to avoid output variability. Rather than treating this variability as a limitation, we show that sampling can produce substantially better solutions than greedy decoding, especially when using reasoning models. We thus propose ThinkTwice, a sampling and selection framework in which the LLM generates multiple candidate templates for a given document, and a selection module chooses the most suitable one. We introduce both an unsupervised method that exploits agreement across generated outputs, and a supervised selection method using reward models trained on labeled DocIE data. To address the scarcity of golden reasoning trajectories for DocIE, we propose a rejection-sampling-based method to generate silver training data that pairs output templates with reasoning traces. Our experiments show the validity of unsupervised and supervised ThinkTwice, consistently outperforming greedy baselines and the state-of-the-art.