Mining Large-Scale Low-Resource Pronunciation Data From Wikipedia
This work addresses the challenge of building speech technology for low-resource languages by providing a publicly available, structured dataset, though it is incremental as it repurposes existing Wikipedia data.
The authors tackled the problem of limited language coverage in grapheme-to-phoneme (G2P) mapping systems by mining pronunciation data from Wikipedia, resulting in a dataset covering 819 languages, including G2P mappings for 63 low-resource languages, 54 of which lacked easily accessible online data.
Pronunciation modeling is a key task for building speech technology in new languages, and while solid grapheme-to-phoneme (G2P) mapping systems exist, language coverage can stand to be improved. The information needed to build G2P models for many more languages can easily be found on Wikipedia, but unfortunately, it is stored in disparate formats. We report on a system we built to mine a pronunciation data set in 819 languages from loosely structured tables within Wikipedia. The data includes phoneme inventories, and for 63 low-resource languages, also includes the grapheme-to-phoneme (G2P) mapping. 54 of these languages do not have easily findable G2P mappings online otherwise. We turned the information from Wikipedia into a structured, machine-readable TSV format, and make the resulting data set publicly available so it can be improved further and used in a variety of applications involving low-resource languages.