Approximating Human-Like Few-shot Learning with GPT-based Compression
This work addresses the challenge of improving few-shot learning capabilities in NLP models for researchers and practitioners, though it is incremental as it builds on existing GPT frameworks.
The authors tackled the problem of enabling generative pre-trained models to perform human-like few-shot learning by framing learning as information compression, using GPT to approximate Kolmogorov complexity and achieve a compression ratio of 15.5 on enwik9 with LLAMA2-7B, and demonstrated superior performance on NLP tasks like semantic similarity and zero-shot classification compared to baselines.
In this work, we conceptualize the learning process as information compression. We seek to equip generative pre-trained models with human-like learning capabilities that enable data compression during inference. We present a novel approach that utilizes the Generative Pre-trained Transformer (GPT) to approximate Kolmogorov complexity, with the aim of estimating the optimal Information Distance for few-shot learning. We first propose using GPT as a prior for lossless text compression, achieving a noteworthy compression ratio. Experiment with LLAMA2-7B backbone achieves a compression ratio of 15.5 on enwik9. We justify the pre-training objective of GPT models by demonstrating its equivalence to the compression length, and, consequently, its ability to approximate the information distance for texts. Leveraging the approximated information distance, our method allows the direct application of GPT models in quantitative text similarity measurements. Experiment results show that our method overall achieves superior performance compared to embedding and prompt baselines on challenging NLP tasks, including semantic similarity, zero and one-shot text classification, and zero-shot text ranking.