Analyzing Syntactic Generalization Capacity of Pre-trained Language Models on Japanese Honorific Conversion
This addresses a domain-specific problem for Japanese language processing, but it is incremental as it applies existing methods to a new linguistic task.
The study tackled the problem of whether pre-trained language models can handle Japanese honorific conversion, which requires grammatical and contextual knowledge, by testing GPT-3 on a dataset with various sentence structures. The result showed that fine-tuned GPT-3 performed better than prompt-based methods, achieving syntactic generalizability except in cases involving direct speech.
Using Japanese honorifics is challenging because it requires not only knowledge of the grammatical rules but also contextual information, such as social relationships. It remains unclear whether pre-trained large language models (LLMs) can flexibly handle Japanese honorifics like humans. To analyze this, we introduce an honorific conversion task that considers social relationships among people mentioned in a conversation. We construct a Japanese honorifics dataset from problem templates of various sentence structures to investigate the syntactic generalization capacity of GPT-3, one of the leading LLMs, on this task under two settings: fine-tuning and prompt learning. Our results showed that the fine-tuned GPT-3 performed better in a context-aware honorific conversion task than the prompt-based one. The fine-tuned model demonstrated overall syntactic generalizability towards compound honorific sentences, except when tested with the data involving direct speech.