Language Modelling as a Multi-Task Problem
This work addresses the intersection of multi-task learning, linguistics, and interpretability, potentially leading to new insights in these domains, but it appears incremental as it builds on existing theories without claiming major breakthroughs.
The paper tackles the problem of understanding language modeling as a multi-task problem by investigating whether language models adhere to multi-task learning principles during training, using Negative Polarity Items as a case study, and finds that a multi-task setting emerges naturally within language modeling objectives.
In this paper, we propose to study language modelling as a multi-task problem, bringing together three strands of research: multi-task learning, linguistics, and interpretability. Based on hypotheses derived from linguistic theory, we investigate whether language models adhere to learning principles of multi-task learning during training. To showcase the idea, we analyse the generalisation behaviour of language models as they learn the linguistic concept of Negative Polarity Items (NPIs). Our experiments demonstrate that a multi-task setting naturally emerges within the objective of the more general task of language modelling.We argue that this insight is valuable for multi-task learning, linguistics and interpretability research and can lead to exciting new findings in all three domains.