Fine-tuning GPT-3 for Russian Text Summarization
This work addresses the lack of specialized summarization solutions for Russian, but it is incremental as it applies an existing method to a new language domain.
The paper tackled the problem of automatic text summarization for the Russian language by fine-tuning GPT-3 on Russian news corpora, achieving state-of-the-art performance without architectural changes, though it noted issues like altering named entities and factual deviations.
Automatic summarization techniques aim to shorten and generalize information given in the text while preserving its core message and the most relevant ideas. This task can be approached and treated with a variety of methods, however, not many attempts have been made to produce solutions specifically for the Russian language despite existing localizations of the state-of-the-art models. In this paper, we aim to showcase ruGPT3 ability to summarize texts, fine-tuning it on the corpora of Russian news with their corresponding human-generated summaries. Additionally, we employ hyperparameter tuning so that the model's output becomes less random and more tied to the original text. We evaluate the resulting texts with a set of metrics, showing that our solution can surpass the state-of-the-art model's performance without additional changes in architecture or loss function. Despite being able to produce sensible summaries, our model still suffers from a number of flaws, namely, it is prone to altering Named Entities present in the original text (such as surnames, places, dates), deviating from facts stated in the given document, and repeating the information in the summary.