Multi-Attribute Multi-Grained Adaptation of Pre-Trained Language Models for Text Understanding from Bayesian Perspective
This work addresses the problem of improving text understanding in NLP by adapting PLMs to handle non-IID data, but it appears incremental as it builds on existing multi-domain and attribute-injecting methods.
The study tackled the unclear impact of non-IID information on pre-trained language models for text understanding by proposing a multi-attribute multi-grained adaptation framework (M2A) from a Bayesian perspective, demonstrating superior performance, especially with larger models and implicitly non-IID data.
Current neural networks often employ multi-domain-learning or attribute-injecting mechanisms to incorporate non-independent and identically distributed (non-IID) information for text understanding tasks by capturing individual characteristics and the relationships among samples. However, the extent of the impact of non-IID information and how these methods affect pre-trained language models (PLMs) remains unclear. This study revisits the assumption that non-IID information enhances PLMs to achieve performance improvements from a Bayesian perspective, which unearths and integrates non-IID and IID features. Furthermore, we proposed a multi-attribute multi-grained framework for PLM adaptations (M2A), which combines multi-attribute and multi-grained views to mitigate uncertainty in a lightweight manner. We evaluate M2A through prevalent text-understanding datasets and demonstrate its superior performance, mainly when data are implicitly non-IID, and PLMs scale larger.