GePpeTto Carves Italian into a Language Model
This work addresses the problem of limited language model availability for non-English languages like Italian, but it is incremental as it applies an existing method to new data.
The authors tackled the lack of generative language models for Italian by developing GePpeTto, the first such model based on GPT-2, and found that its output was judged as natural more often than not in human evaluations, with performance closer to human texts than a baseline model.
In the last few years, pre-trained neural architectures have provided impressive improvements across several NLP tasks. Still, generative language models are available mainly for English. We develop GePpeTto, the first generative language model for Italian, built using the GPT-2 architecture. We provide a thorough analysis of GePpeTto's quality by means of both an automatic and a human-based evaluation. The automatic assessment consists in (i) calculating perplexity across different genres and (ii) a profiling analysis over GePpeTto's writing characteristics. We find that GePpeTto's production is a sort of bonsai version of human production, with shorter but yet complex sentences. Human evaluation is performed over a sentence completion task, where GePpeTto's output is judged as natural more often than not, and much closer to the original human texts than to a simpler language model which we take as baseline.