Efficient Attribute Injection for Pretrained Language Models
This addresses the challenge of attribute injection in PLMs for NLP tasks, offering a practical solution for domains with large attribute vocabularies or sparse/multi-labeled attributes, though it is incremental as it builds on existing adapter techniques.
The paper tackles the problem of efficiently incorporating metadata attributes into pretrained language models (PLMs) by proposing a lightweight method using extended adapters with low-rank approximations and hypercomplex multiplications, which outperforms previous methods and achieves state-of-the-art performance on eight datasets from different domains.
Metadata attributes (e.g., user and product IDs from reviews) can be incorporated as additional inputs to neural-based NLP models, by modifying the architecture of the models, in order to improve their performance. Recent models however rely on pretrained language models (PLMs), where previously used techniques for attribute injection are either nontrivial or ineffective. In this paper, we propose a lightweight and memory-efficient method to inject attributes to PLMs. We extend adapters, i.e. tiny plug-in feed-forward modules, to include attributes both independently of or jointly with the text. To limit the increase of parameters especially when the attribute vocabulary is large, we use low-rank approximations and hypercomplex multiplications, significantly decreasing the total parameters. We also introduce training mechanisms to handle domains in which attributes can be multi-labeled or sparse. Extensive experiments and analyses on eight datasets from different domains show that our method outperforms previous attribute injection methods and achieves state-of-the-art performance on various datasets.