Once Upon a Time: Interactive Learning for Storytelling with Small Language Models
This addresses data efficiency in language model training, particularly for storytelling, but is incremental as it builds on existing interactive learning methods.
The paper tackles the problem of training language models with less data by incorporating high-level, cognitively inspired feedback from a teacher model, showing that interactive learning with just 1 million words can improve storytelling skills as much as 410 million words of next-word prediction.
Children efficiently acquire language not just by listening, but by interacting with others in their social environment. Conversely, large language models are typically trained with next-word prediction on massive amounts of text. Motivated by this contrast, we investigate whether language models can be trained with less data by learning not only from next-word prediction but also from high-level, cognitively inspired feedback. We train a student model to generate stories, which a teacher model rates on readability, narrative coherence, and creativity. By varying the amount of pretraining before the feedback loop, we assess the impact of this interactive learning on formal and functional linguistic competence. We find that the high-level feedback is highly data efficient: With just 1 M words of input in interactive learning, storytelling skills can improve as much as with 410 M words of next-word prediction.