Exploring the Effectiveness of Methods for Persona Extraction
This work addresses persona extraction for dialogue systems in Russian, but it is incremental as it applies existing methods to a new language with minor improvements.
The paper studied methods for extracting dialogue participant personas in Russian, finding that all tested models had insufficient recall, but using NCE Loss improved precision and larger models enhanced persona extraction.
The paper presents a study of methods for extracting information about dialogue participants and evaluating their performance in Russian. To train models for this task, the Multi-Session Chat dataset was translated into Russian using multiple translation models, resulting in improved data quality. A metric based on the F-score concept is presented to evaluate the effectiveness of the extraction models. The metric uses a trained classifier to identify the dialogue participant to whom the persona belongs. Experiments were conducted on MBart, FRED-T5, Starling-7B, which is based on the Mistral, and Encoder2Encoder models. The results demonstrated that all models exhibited an insufficient level of recall in the persona extraction task. The incorporation of the NCE Loss improved the model's precision at the expense of its recall. Furthermore, increasing the model's size led to enhanced extraction of personas.