ICL CIPHERS: Quantifying "Learning" in In-Context Learning via Substitution Ciphers
This provides a novel approach to quantify 'learning' in ICL, addressing a challenge in understanding LLM behavior, though it is incremental in scope.
The paper tackled the problem of disentangling task retrieval from task learning in in-context learning (ICL) by introducing ICL CIPHERS, a method using substitution ciphers to reformulate tasks, and found that LLMs perform better with bijective ciphers than non-bijective ones, showing a small but consistent gap across four datasets and six models.
Recent works have suggested that In-Context Learning (ICL) operates in dual modes, i.e. task retrieval (remember learned patterns from pre-training) and task learning (inference-time ''learning'' from demonstrations). However, disentangling these the two modes remains a challenging goal. We introduce ICL CIPHERS, a class of task reformulations based on substitution ciphers borrowed from classic cryptography. In this approach, a subset of tokens in the in-context inputs are substituted with other (irrelevant) tokens, rendering English sentences less comprehensible to human eye. However, by design, there is a latent, fixed pattern to this substitution, making it reversible. This bijective (reversible) cipher ensures that the task remains a well-defined task in some abstract sense, despite the transformations. It is a curious question if LLMs can solve tasks reformulated by ICL CIPHERS with a BIJECTIVE mapping, which requires ''deciphering'' the latent cipher. We show that LLMs are better at solving tasks reformulated by ICL CIPHERS with BIJECTIVE mappings than the NON-BIJECTIVE (irreversible) baseline, providing a novel approach to quantify ''learning'' in ICL. While this gap is small, it is consistent across the board on four datasets and six models. Finally, we examine LLMs' internal representations and identify evidence in their ability to decode the ciphered inputs.