X-PARADE: Cross-Lingual Textual Entailment and Information Divergence across Paragraphs
This addresses the challenge of cross-lingual textual entailment and fact-checking for NLP applications, but it is incremental as it builds on existing datasets and methods.
The paper tackled the problem of cross-lingual information divergence at the paragraph level by introducing X-PARADE, the first dataset for this task, and found that various methods, including token alignment and LLM prompting, all performed below human levels.
Understanding when two pieces of text convey the same information is a goal touching many subproblems in NLP, including textual entailment and fact-checking. This problem becomes more complex when those two pieces of text are in different languages. Here, we introduce X-PARADE (Cross-lingual Paragraph-level Analysis of Divergences and Entailments), the first cross-lingual dataset of paragraph-level information divergences. Annotators label a paragraph in a target language at the span level and evaluate it with respect to a corresponding paragraph in a source language, indicating whether a given piece of information is the same, new, or new but can be inferred. This last notion establishes a link with cross-language NLI. Aligned paragraphs are sourced from Wikipedia pages in different languages, reflecting real information divergences observed in the wild. Armed with our dataset, we investigate a diverse set of approaches for this problem, including token alignment from machine translation, textual entailment methods that localize their decisions, and prompting LLMs. Our results show that these methods vary in their capability to handle inferable information, but they all fall short of human performance.