Emergent Semantic Role Understanding in Language Models
For NLP researchers, this work clarifies the extent to which linguistic structure (semantic roles) is learned from unsupervised pre-training versus supervised fine-tuning.
The study investigates whether semantic role understanding emerges during language model pre-training or requires task-specific fine-tuning. Using linear probes on frozen decoder-only transformers, they find that pre-trained representations encode substantial semantic role information, but fine-tuning still improves performance, indicating partial emergence.
Understanding how linguistic structure emerges in language models is central to interpreting what these systems learn from data and how much supervision they truly require. In particular, semantic role understanding ("who did what to whom") is a core component of meaning representation, yet it remains unclear whether it arises from pre-training alone or depends on task-specific fine-tuning. We study whether semantic role understanding emerges during language model pre-training or requires task-specific fine-tuning. We freeze decoder-only transformers and train linear probes to extract semantic roles, using performance to infer whether role information is already encoded in pre-training or learned during adaptation. Across model scales, we find that frozen representations contain substantial semantic role information, with performance improving but not fully matching fine-tuned models. This indicates partial but incomplete emergence from pre-training alone. We show that semantic role structure emerges from language modeling objectives, but its internal implementation shifts toward more distributed representations as model scale increases.