Prompt Algebra for Task Composition
This work addresses the challenge of task composition for users of Visual Language Models, offering a novel approach with specific performance gains, though it is incremental in nature.
The paper tackles the problem of composing tasks in Visual Language Models by introducing prompt algebra, which combines independently learned prompts, and proposes constrained prompt tuning to enhance composite classifier performance. The method achieves classification accuracy within 2.5% of the best base model on average and improves by 8.45% on UTZappos.
We investigate whether prompts learned independently for different tasks can be later combined through prompt algebra to obtain a model that supports composition of tasks. We consider Visual Language Models (VLM) with prompt tuning as our base classifier and formally define the notion of prompt algebra. We propose constrained prompt tuning to improve performance of the composite classifier. In the proposed scheme, prompts are constrained to appear in the lower dimensional subspace spanned by the basis vectors of the pre-trained vocabulary. Further regularization is added to ensure that the learned prompt is grounded correctly to the existing pre-trained vocabulary. We demonstrate the effectiveness of our method on object classification and object-attribute classification datasets. On average, our composite model obtains classification accuracy within 2.5% of the best base model. On UTZappos it improves classification accuracy over the best base model by 8.45% on average.