KS-Lottery: Finding Certified Lottery Tickets for Multilingual Language Models
This addresses the challenge of parameter-efficient tuning for large language models, offering a method to reduce computational costs while maintaining performance, though it is incremental as it builds on existing lottery ticket hypothesis work.
The paper tackles the problem of finding efficient subsets of parameters for fine-tuning multilingual language models, proposing KS-Lottery to identify certified winning tickets using the Kolmogorov-Smirnov Test, and shows that fine-tuning just 18 tokens' embeddings in LLaMA achieves comparable performance to full fine-tuning on translation tasks.
The lottery ticket hypothesis posits the existence of ``winning tickets'' within a randomly initialized neural network. Do winning tickets exist for LLMs in fine-tuning scenarios? How can we find such winning tickets? In this paper, we propose KS-Lottery, a method to identify a small subset of LLM parameters highly effective in multilingual fine-tuning. Our key idea is to use Kolmogorov-Smirnov Test to analyze the distribution shift of parameters before and after fine-tuning. We further theoretically prove that KS-Lottery can find the certified winning tickets in the embedding layer, fine-tuning on the found parameters is guaranteed to perform as well as full fine-tuning. Comparing KS-Lottery with other parameter-efficient tuning algorithms on translation tasks, the experimental results show that KS-Lottery finds a much smaller set of parameters for fine-tuning while achieving the comparable performance as full fine-tuning LLM. Surprisingly, we find that fine-tuning 18 tokens' embedding of LLaMA suffices to reach the fine-tuning translation performance~\footnote{https://github.com/CONE-MT/KS-Lottery.}.