Less is More: A Lightweight and Robust Neural Architecture for Discourse Parsing
This work addresses overfitting issues in discourse parsing for NLP researchers, offering a more robust and efficient solution, though it is incremental as it builds on existing pretrained language models.
The paper tackles the problem of overfitting in discourse parsing by proposing a lightweight neural architecture that replaces complex feature extractors with learnable self-attention modules, achieving comparable or better performance with fewer parameters and less processing time on three common tasks.
Complex feature extractors are widely employed for text representation building. However, these complex feature extractors make the NLP systems prone to overfitting especially when the downstream training datasets are relatively small, which is the case for several discourse parsing tasks. Thus, we propose an alternative lightweight neural architecture that removes multiple complex feature extractors and only utilizes learnable self-attention modules to indirectly exploit pretrained neural language models, in order to maximally preserve the generalizability of pre-trained language models. Experiments on three common discourse parsing tasks show that powered by recent pretrained language models, the lightweight architecture consisting of only two self-attention layers obtains much better generalizability and robustness. Meanwhile, it achieves comparable or even better system performance with fewer learnable parameters and less processing time.