Strategies for Span Labeling with Large Language Models
This addresses span labeling inconsistencies for users of generative LLMs in text analysis tasks, representing an incremental improvement.
The paper tackled the problem of inconsistent span labeling with large language models by introducing LogitMatch, a constrained decoding method that forces output alignment with valid input spans, improving upon competitive matching-based methods in some setups.
Large language models (LLMs) are increasingly used for text analysis tasks, such as named entity recognition or error detection. Unlike encoder-based models, however, generative architectures lack an explicit mechanism to refer to specific parts of their input. This leads to a variety of ad-hoc prompting strategies for span labeling, often with inconsistent results. In this paper, we categorize these strategies into three families: tagging the input text, indexing numerical positions of spans, and matching span content. To address the limitations of content matching, we introduce LogitMatch, a new constrained decoding method that forces the model's output to align with valid input spans. We evaluate all methods across four diverse tasks. We find that while tagging remains a robust baseline, LogitMatch improves upon competitive matching-based methods by eliminating span matching issues and outperforms other strategies in some setups.