Multiple Choice Learning of Low Rank Adapters for Language Modeling
This addresses the ambiguity in language modeling for applications like captioning, though it appears incremental as it builds on existing methods like LoRA and MCL.
The paper tackles the problem of generating diverse and plausible sentence continuations in language modeling by proposing LoRA-MCL, a training scheme that extends next-token prediction with Multiple Choice Learning and Winner-Takes-All loss. It demonstrates high diversity and relevance in outputs on real-world visual and audio captioning tasks.
We propose LoRA-MCL, a training scheme that extends next-token prediction in language models with a method designed to decode diverse, plausible sentence continuations at inference time. Traditional language modeling is an intrinsically ill-posed problem: given a context, multiple futures may be equally plausible. Our approach leverages Multiple Choice Learning (MCL) and the Winner-Takes-All (WTA) loss to efficiently handle ambiguity through Low-Rank Adaptation (LoRA). We provide a theoretical interpretation of applying Multiple Choice Learning to Language Modeling, assuming the data is generated from a mixture of distributions. To illustrate the proposed approach, we use data sampled from mixtures of Markov chains. We then demonstrate with extensive experiments on real-world visual and audio captioning tasks that our method achieves high diversity and relevance in generated outputs.