minicons: Enabling Flexible Behavioral and Representational Analyses of Transformer Language Models
This tool addresses the need for flexible analysis methods for researchers studying transformer language models, though it is incremental as it builds on existing analysis techniques.
The authors introduced minicons, an open-source library that provides a standard API for behavioral and representational analyses of transformer language models, enabling efficient extraction of probabilities and vectors, and applied it to case studies on BERT's learning dynamics and benchmarking 23 LMs on zero-shot abductive reasoning.
We present minicons, an open source library that provides a standard API for researchers interested in conducting behavioral and representational analyses of transformer-based language models (LMs). Specifically, minicons enables researchers to apply analysis methods at two levels: (1) at the prediction level -- by providing functions to efficiently extract word/sentence level probabilities; and (2) at the representational level -- by also facilitating efficient extraction of word/phrase level vectors from one or more layers. In this paper, we describe the library and apply it to two motivating case studies: One focusing on the learning dynamics of the BERT architecture on relative grammatical judgments, and the other on benchmarking 23 different LMs on zero-shot abductive reasoning. minicons is available at https://github.com/kanishkamisra/minicons