A multilabel approach to morphosyntactic probing
This work addresses the need for better evaluation of linguistic properties in multilingual models, though it is incremental as it builds on existing probing methods.
The paper tackled the problem of assessing morphosyntactic representations in multilingual language models by introducing a multilabel probing task, demonstrating it with multilingual BERT across seven diverse languages and showing that many features are easily extractable, with probes performing well on nouns in zero-shot transfer to six held-out languages.
We introduce a multilabel probing task to assess the morphosyntactic representations of word embeddings from multilingual language models. We demonstrate this task with multilingual BERT (Devlin et al., 2018), training probes for seven typologically diverse languages of varying morphological complexity: Afrikaans, Croatian, Finnish, Hebrew, Korean, Spanish, and Turkish. Through this simple but robust paradigm, we show that multilingual BERT renders many morphosyntactic features easily and simultaneously extractable (e.g., gender, grammatical case, pronominal type). We further evaluate the probes on six "held-out" languages in a zero-shot transfer setting: Arabic, Chinese, Marathi, Slovenian, Tagalog, and Yoruba. This style of probing has the added benefit of revealing the linguistic properties that language models recognize as being shared across languages. For instance, the probes performed well on recognizing nouns in the held-out languages, suggesting that multilingual BERT has a conception of noun-hood that transcends individual languages; yet, the same was not true of adjectives.