Towards Efficient Post-Training via Fourier-Driven Adapter Architectures
This addresses the need for efficient adaptation of large language models with minimal computational cost, though it appears incremental as it builds on existing adapter methods.
The paper tackles the problem of parameter-efficient fine-tuning for large pre-trained language models by proposing the Fourier-Activated Adapter (FAA) framework, which uses random Fourier features to enable frequency-aware modulation and achieves competitive or superior performance on benchmarks like GLUE and E2E NLG while maintaining low overhead.
We propose a novel framework, termed Fourier-Activated Adapter (FAA), for parameter-efficient fine-tuning of large pre-trained language models. By incorporating random Fourier features into lightweight adapter modules, FAA decomposes intermediate representations into complementary low- and high-frequency components, enabling frequency-aware modulation of semantic information. This design allows the model to selectively emphasize informative frequency bands during adaptation while preserving the representational capacity of the frozen backbone. Extensive experiments on GLUE, E2E NLG, and instruction-tuning benchmarks demonstrate that FAA consistently achieves competitive or superior performance compared to existing parameter-efficient fine-tuning methods, while maintaining low computational and memory overhead. Ablation studies further verify the effectiveness of frequency-aware activation and adaptive weighting mechanisms, highlighting FAA as a robust and efficient approach for post-training large language models.