Do Transformers Encode a Foundational Ontology? Probing Abstract Classes in Natural Language
This work addresses the problem of understanding the semantic capabilities of AI models for researchers in natural language processing, though it is incremental as it extends existing probing methods to a more abstract level.
The paper investigates whether Transformer-based models encode abstract semantic information related to Foundational Ontologies, finding that they incidentally encode such information during pre-training and can be used to build robust taggers with 90% accuracy.
With the methodological support of probing (or diagnostic classification), recent studies have demonstrated that Transformers encode syntactic and semantic information to some extent. Following this line of research, this paper aims at taking semantic probing to an abstraction extreme with the goal of answering the following research question: can contemporary Transformer-based models reflect an underlying Foundational Ontology? To this end, we present a systematic Foundational Ontology (FO) probing methodology to investigate whether Transformers-based models encode abstract semantic information. Following different pre-training and fine-tuning regimes, we present an extensive evaluation of a diverse set of large-scale language models over three distinct and complementary FO tagging experiments. Specifically, we present and discuss the following conclusions: (1) The probing results indicate that Transformer-based models incidentally encode information related to Foundational Ontologies during the pre-training pro-cess; (2) Robust FO taggers (accuracy of 90 percent)can be efficiently built leveraging on this knowledge.