Conditional Adapters: Parameter-efficient Transfer Learning with Fast Inference
This addresses the need for faster and more efficient model adaptation in machine learning applications, though it appears incremental as it builds on existing adapter methods.
The paper tackles the problem of parameter-efficient transfer learning with slow inference by proposing Conditional Adapters (CoDA), which achieve a 2x to 8x inference speed-up compared to state-of-the-art Adapter approaches with moderate to no accuracy loss across language, vision, and speech tasks.
We propose Conditional Adapter (CoDA), a parameter-efficient transfer learning method that also improves inference efficiency. CoDA generalizes beyond standard adapter approaches to enable a new way of balancing speed and accuracy using conditional computation. Starting with an existing dense pretrained model, CoDA adds sparse activation together with a small number of new parameters and a light-weight training phase. Our experiments demonstrate that the CoDA approach provides an unexpectedly efficient way to transfer knowledge. Across a variety of language, vision, and speech tasks, CoDA achieves a 2x to 8x inference speed-up compared to the state-of-the-art Adapter approaches with moderate to no accuracy loss and the same parameter efficiency.