FlowNIB: An Information Bottleneck Analysis of Bidirectional vs. Unidirectional Language Models
This provides a principled explanation for the effectiveness of bidirectional architectures in natural language understanding, addressing a key theoretical problem for researchers in machine learning and NLP.
The paper tackled the theoretical gap in why bidirectional language models outperform unidirectional ones by applying the Information Bottleneck principle, showing that bidirectional models retain more mutual information and have higher effective dimensionality, with experimental validation across multiple tasks.
Bidirectional language models have better context understanding and perform better than unidirectional models on natural language understanding tasks, yet the theoretical reasons behind this advantage remain unclear. In this work, we investigate this disparity through the lens of the Information Bottleneck (IB) principle, which formalizes a trade-off between compressing input information and preserving task-relevant content. We propose FlowNIB, a dynamic and scalable method for estimating mutual information during training that addresses key limitations of classical IB approaches, including computational intractability and fixed trade-off schedules. Theoretically, we show that bidirectional models retain more mutual information and exhibit higher effective dimensionality than unidirectional models. To support this, we present a generalized framework for measuring representational complexity and prove that bidirectional representations are strictly more informative under mild conditions. We further validate our findings through extensive experiments across multiple models and tasks using FlowNIB, revealing how information is encoded and compressed throughout training. Together, our work provides a principled explanation for the effectiveness of bidirectional architectures and introduces a practical tool for analyzing information flow in deep language models.