Example-based Hypernetworks for Out-of-Distribution Generalization
This addresses the challenge of multi-source adaptation for unfamiliar domains in NLP, offering a novel approach that could enhance model robustness in real-world applications.
The paper tackles out-of-distribution generalization in NLP by using example-based Hypernetworks to adapt from multiple source domains to unknown target domains, achieving superior performance over established algorithms in 29 adaptation scenarios for tasks like sentiment classification and natural language inference.
As Natural Language Processing (NLP) algorithms continually achieve new milestones, out-of-distribution generalization remains a significant challenge. This paper addresses the issue of multi-source adaptation for unfamiliar domains: We leverage labeled data from multiple source domains to generalize to unknown target domains at training. Our innovative framework employs example-based Hypernetwork adaptation: a T5 encoder-decoder initially generates a unique signature from an input example, embedding it within the source domains' semantic space. This signature is subsequently utilized by a Hypernetwork to generate the task classifier's weights. We evaluated our method across two tasks - sentiment classification and natural language inference - in 29 adaptation scenarios, where it outpaced established algorithms. In an advanced version, the signature also enriches the input example's representation. We also compare our finetuned architecture to few-shot GPT-3, demonstrating its effectiveness in essential use cases. To our knowledge, this marks the first application of Hypernetworks to the adaptation for unknown domains.