PARA: Parameter-Efficient Fine-tuning with Prompt Aware Representation Adjustment
This addresses the industry demand for efficient and high-performance PEFT methods in multi-tenant settings, representing an incremental improvement over existing techniques like LoRA.
The paper tackles the problem of parameter-efficient fine-tuning (PEFT) for single-backbone multi-tenant applications by introducing PARA, a method that integrates a lightweight vector generator in Transformer layers to adjust hidden representations based on input prompts, resulting in performance surpassing current PEFT benchmarks with similar parameter counts and greater efficiency than LoRA.
In the realm of parameter-efficient fine-tuning (PEFT) methods, while options like LoRA are available, there is a persistent demand in the industry for a PEFT approach that excels in both efficiency and performance within the context of single-backbone multi-tenant applications. This paper introduces a new and straightforward PEFT technique, termed \underline{P}rompt \underline{A}ware \underline{R}epresentation \underline{A}djustment (PARA). The core of our proposal is to integrate a lightweight vector generator within each Transformer layer. This generator produces vectors that are responsive to input prompts, thereby adjusting the hidden representations accordingly. Our extensive experimentation across diverse tasks has yielded promising results. Firstly, the PARA method has been shown to surpass current PEFT benchmarks in terms of performance, despite having a similar number of adjustable parameters. Secondly, it has proven to be more efficient than LoRA in the single-backbone multi-tenant scenario, highlighting its significant potential for industrial adoption.