STARE at the Structure: Steering ICL Exemplar Selection with Structural Alignment
This work addresses a key bottleneck in in-context learning for structured prediction tasks, offering an incremental improvement in exemplar selection efficiency and performance.
The paper tackles the problem of suboptimal exemplar selection in in-context learning for structured prediction tasks like semantic parsing by proposing a two-stage strategy that uses structure-aware supervision and a plug-in module to enhance structural alignment. The method consistently outperforms existing baselines across four benchmarks and three semantic parsing tasks with multiple large language models.
In-Context Learning (ICL) has become a powerful paradigm that enables LLMs to perform a wide range of tasks without task-specific fine-tuning. However, the effectiveness of ICL heavily depends on the quality of exemplar selection. In particular, for structured prediction tasks such as semantic parsing, existing ICL selection strategies often overlook structural alignment, leading to suboptimal performance and poor generalization. To address this issue, we propose a novel two-stage exemplar selection strategy that achieves a strong balance between efficiency, generalizability, and performance. First, we fine-tune a BERT-based retriever using structure-aware supervision, guiding it to select exemplars that are both semantically relevant and structurally aligned. Then, we enhance the retriever with a plug-in module, which amplifies syntactically meaningful information in the hidden representations. This plug-in is model-agnostic, requires minimal overhead, and can be seamlessly integrated into existing pipelines. Experiments on four benchmarks spanning three semantic parsing tasks demonstrate that our method consistently outperforms existing baselines with multiple recent LLMs as inference-time models.