Can LLMs Extract Frame-Semantic Arguments?
It addresses a critical but underexplored task in natural language understanding for researchers and practitioners, though it is incremental as it builds on existing LLM capabilities.
This paper tackled the problem of evaluating large language models (LLMs) for extracting frame-semantic arguments, revealing that JSON-based representations boost performance and a novel approach achieves state-of-the-art results on ambiguous targets, with models from 0.5B to 78B parameters showing better performance with size but smaller ones can be competitive through fine-tuning.
Frame-semantic parsing is a critical task in natural language understanding, yet the ability of large language models (LLMs) to extract frame-semantic arguments remains underexplored. This paper presents a comprehensive evaluation of LLMs on frame-semantic argument identification, analyzing the impact of input representation formats, model architectures, and generalization to unseen and out-of-domain samples. Our experiments, spanning models from 0.5B to 78B parameters, reveal that JSON-based representations significantly enhance performance, and while larger models generally perform better, smaller models can achieve competitive results through fine-tuning. We also introduce a novel approach to frame identification leveraging predicted frame elements, achieving state-of-the-art performance on ambiguous targets. Despite strong generalization capabilities, our analysis finds that LLMs still struggle with out-of-domain data.