Harvesting comparable corpora and mining them for equivalent bilingual sentences using statistical classification and analogy- based heuristics
This addresses the scarcity of parallel data for cross-lingual applications like machine translation, offering incremental improvements by leveraging more abundant non-parallel multilingual resources.
The paper tackles the problem of mining parallel sentences from comparable corpora, proposing web crawling and analogy-based methods to build subject-aligned corpora, resulting in improvements in machine translation for Polish-English across various domains.
Parallel sentences are a relatively scarce but extremely useful resource for many applications including cross-lingual retrieval and statistical machine translation. This research explores our new methodologies for mining such data from previously obtained comparable corpora. The task is highly practical since non-parallel multilingual data exist in far greater quantities than parallel corpora, but parallel sentences are a much more useful resource. Here we propose a web crawling method for building subject-aligned comparable corpora from e.g. Wikipedia dumps and Euronews web page. The improvements in machine translation are shown on Polish-English language pair for various text domains. We also tested another method of building parallel corpora based on comparable corpora data. It lets automatically broad existing corpus of sentences from subject of corpora based on analogies between them.