Investigating the Effectiveness of Task-Agnostic Prefix Prompt for Instruction Following
This addresses the challenge of enhancing instruction-following ability in LLMs during inference, offering a simple, reproducible method for zero-shot generalization.
The paper tackles the problem of improving instruction-following in Large Language Models by prepending a Task-Agnostic Prefix Prompt (TAPP) to inputs, resulting in average improvements of 34.58% for base models and 12.26% for instruction-tuned models.
In this paper, we present our finding that prepending a Task-Agnostic Prefix Prompt (TAPP) to the input improves the instruction-following ability of various Large Language Models (LLMs) during inference. TAPP is different from canonical prompts for LLMs in that it is a fixed prompt prepended to the beginning of every input regardless of the target task for zero-shot generalization. We observe that both base LLMs (i.e. not fine-tuned to follow instructions) and instruction-tuned models benefit from TAPP, resulting in 34.58% and 12.26% improvement on average, respectively. This implies that the instruction-following ability of LLMs can be improved during inference time with a fixed prompt constructed with simple heuristics. We hypothesize that TAPP assists language models to better estimate the output distribution by focusing more on the instruction of the target task during inference. In other words, such ability does not seem to be sufficiently activated in not only base LLMs but also many instruction-fine-tuned LLMs. All experiments are reproducible from https://github.com/seonghyeonye/TAPP.