Multilingual Open Text Release 1: Public Domain News in 44 Languages
This provides a valuable dataset for natural language processing in many low-resource languages, though it is incremental as it builds on existing public domain sources.
The authors tackled the lack of multilingual text resources by releasing Multilingual Open Text (MOT), a corpus with over 2.8 million news articles and 1 million snippets in 44 languages, collected from Voice of America's public domain news.
We present Multilingual Open Text (MOT), a new multilingual corpus containing text in 44 languages, many of which have limited existing text resources for natural language processing. The first release of the corpus contains over 2.8 million news articles and an additional 1 million short snippets (photo captions, video descriptions, etc.) published between 2001--2022 and collected from Voice of America's news websites. We describe our process for collecting, filtering, and processing the data. The source material is in the public domain, our collection is licensed using a creative commons license (CC BY 4.0), and all software used to create the corpus is released under the MIT License. The corpus will be regularly updated as additional documents are published.