Reduction of Parameter Redundancy in Biaffine Classifiers with Symmetric and Circulant Weight Matrices
This work addresses overfitting issues in dependency parsing for NLP researchers, but it is incremental as it builds on existing biaffine classifiers.
The paper tackled the problem of parameter redundancy in biaffine classifiers, which can cause overfitting, by assuming symmetric or circulant weight matrices, resulting in over 16% parameter reduction while achieving better or comparable accuracy on most treebanks in the CoNLL 2017 dataset.
Currently, the biaffine classifier has been attracting attention as a method to introduce an attention mechanism into the modeling of binary relations. For instance, in the field of dependency parsing, the Deep Biaffine Parser by Dozat and Manning has achieved state-of-the-art performance as a graph-based dependency parser on the English Penn Treebank and CoNLL 2017 shared task. On the other hand, it is reported that parameter redundancy in the weight matrix in biaffine classifiers, which has O(n^2) parameters, results in overfitting (n is the number of dimensions). In this paper, we attempted to reduce the parameter redundancy by assuming either symmetry or circularity of weight matrices. In our experiments on the CoNLL 2017 shared task dataset, our model achieved better or comparable accuracy on most of the treebanks with more than 16% parameter reduction.