NCL-UoR at SemEval-2026 Task 5: Embedding-Based Methods, Fine-Tuning, and LLMs for Word Sense Plausibility Rating
For researchers in semantic plausibility and narrative understanding, this work provides a systematic comparison showing that structured LLM prompting can surpass fine-tuned models on a specific rating task.
The paper compares embedding-based methods, fine-tuning, and LLM prompting for word sense plausibility rating on a 1-5 scale. The best system uses structured prompting with decision rules, outperforming other approaches, and shows that prompt design matters more than model scale.
Word sense plausibility rating requires predicting the human-perceived plausibility of a given word sense on a 1-5 scale in the context of short narrative stories containing ambiguous homonyms. This paper systematically compares three approaches: (1) embedding-based methods pairing sentence embeddings with standard regressors, (2) transformer fine-tuning with parameter-efficient adaptation, and (3) large language model (LLM) prompting with structured reasoning and explicit decision rules. The best-performing system employs a structured prompting strategy that decomposes evaluation into narrative components (precontext, target sentence, ending) and applies explicit decision rules for rating calibration. The analysis reveals that structured prompting with decision rules outperforms both fine-tuned models and embedding-based approaches, and that prompt design matters more than model scale for this task.