Breaking Memorization Barriers in LLM Code Fine-Tuning via Information Bottleneck for Improved Generalization
This addresses a failure mode in code fine-tuning for developers, offering a method to enhance generalization, though it is incremental as it builds on existing fine-tuning approaches.
The paper tackles the memorization barrier in fine-tuning LLMs for code generation, where strong memorization of downstream data hinders learning of generalizable code knowledge, and proposes an information bottleneck-guided fine-tuning method that improves top-1 performance and stability on code benchmarks.
Adapting pretrained large language models (LLMs) to code domains via supervised fine-tuning (FT) has been commonly used for code generation. However, we identify a previously underappreciated failure mode, the memorization barrier, where strong memorization of downstream code data in the base model could trap optimization and prevent the standard FT from effectively acquiring new, generalizable code knowledge. To overcome this barrier, we propose the information bottleneck (IB)-guided fine-tuning, termed IB-FT, which applies an IB penalty on hidden representations of the code data to compress spurious, memorized features while preserving task-relevant information. Extensive experiments on two code benchmarks (OriGen and Evol-CodeAlpaca-V1) show that IB-FT substantially alleviates the memorization barrier, improves top-1 performance (Pass@$1$), and yields far more stable gains under the stricter multi-sample metric Pass@$k^{(m)}$ (a problem counts as solved only if at least $m$ of $k$ samples pass unit tests) compared with conventional FT.