A Matter of Framing: The Impact of Linguistic Formalism on Probing Results
This work highlights a critical methodological consideration for researchers in NLP probing studies, showing that formalism choice can impact results, though it is incremental in nature.
The study investigated how the choice of linguistic formalism affects probing results in BERT models, finding that formalism leads to meaningful differences in encoding semantic role information and that layer probing can detect subtle variations between implementations of the same formalism.
Deep pre-trained contextualized encoders like BERT (Delvin et al., 2019) demonstrate remarkable performance on a range of downstream tasks. A recent line of research in probing investigates the linguistic knowledge implicitly learned by these models during pre-training. While most work in probing operates on the task level, linguistic tasks are rarely uniform and can be represented in a variety of formalisms. Any linguistics-based probing study thereby inevitably commits to the formalism used to annotate the underlying data. Can the choice of formalism affect probing results? To investigate, we conduct an in-depth cross-formalism layer probing study in role semantics. We find linguistically meaningful differences in the encoding of semantic role- and proto-role information by BERT depending on the formalism and demonstrate that layer probing can detect subtle differences between the implementations of the same linguistic formalism. Our results suggest that linguistic formalism is an important dimension in probing studies, along with the commonly used cross-task and cross-lingual experimental settings.