Understanding How Model Size Affects Few-shot Instruction Prompting
This addresses the problem of optimizing few-shot learning for large language models, which is incremental as it builds on existing scaling studies.
The paper investigates how model size influences few-shot instruction prompting, finding that accuracy on a word discrimination task degrades with larger models under extremely few-shot conditions, but larger models benefit more from additional examples.
Large Language Models are affected by the phenomena of memorizing and forgetting their training data. But how do these vary by model size? We work towards this question by investigating how the model size affects the model's ability to discriminate a word's meaning in a given context. We introduce a dataset called DeltaWords, which evaluates a model's ability to follow instructions to select a sentence which replaces the target word with its antonym. We show a weak inverse scaling trend, where task accuracy degrades as model size increase, under extremely few-shot prompting regimes. We show that increasing the number of examples tend to disproportionately benefit larger models than smaller models.