Noise-Robustness Through Noise: A Framework combining Asymmetric LoRA with Poisoning MoE
This addresses noise susceptibility in fine-tuning for NLP applications, offering a low-cost alternative to data cleaning, though it is incremental as it builds on existing LoRA and MoE methods.
The paper tackles the problem of noisy data interference in parameter-efficient fine-tuning of pre-trained language models by proposing LoPE, a framework that uses generated noisy data to enhance robustness, achieving strong performance without data cleaning.
Current parameter-efficient fine-tuning methods for adapting pre-trained language models to downstream tasks are susceptible to interference from noisy data. Conventional noise-handling approaches either rely on laborious data pre-processing or employ model architecture modifications prone to error accumulation. In contrast to existing noise-process paradigms, we propose a noise-robust adaptation method via asymmetric LoRA poisoning experts (LoPE), a novel framework that enhances model robustness to noise only with generated noisy data. Drawing inspiration from the mixture-of-experts architecture, LoPE strategically integrates a dedicated poisoning expert in an asymmetric LoRA configuration. Through a two-stage paradigm, LoPE performs noise injection on the poisoning expert during fine-tuning to enhance its noise discrimination and processing ability. During inference, we selectively mask the dedicated poisoning expert to leverage purified knowledge acquired by normal experts for noise-robust output. Extensive experiments demonstrate that LoPE achieves strong performance and robustness purely through the low-cost noise injection, which completely eliminates the requirement of data cleaning.