Better Long-Range Dependency By Bootstrapping A Mutual Information Regularizer
This addresses the challenge of capturing complex long-range dependencies in sequences, such as in language modeling, with an incremental improvement over existing methods.
The paper tackles the problem of learning long-range dependencies in sequence data by introducing a mutual information regularizer, which improves language model likelihood on holdout data and generation quality.
In this work, we develop a novel regularizer to improve the learning of long-range dependency of sequence data. Applied on language modelling, our regularizer expresses the inductive bias that sequence variables should have high mutual information even though the model might not see abundant observations for complex long-range dependency. We show how the `next sentence prediction (classification)' heuristic can be derived in a principled way from our mutual information estimation framework, and be further extended to maximize the mutual information of sequence variables. The proposed approach not only is effective at increasing the mutual information of segments under the learned model but more importantly, leads to a higher likelihood on holdout data, and improved generation quality. Code is released at https://github.com/BorealisAI/BMI.