FineFreq: A Multilingual Character Frequency Dataset from Web-Scale Text
This provides a resource for researchers and practitioners in natural language processing and linguistics working with multilingual text analysis, though it is incremental as it builds on existing corpora.
The authors tackled the problem of lacking large-scale multilingual character frequency data by creating FineFreq, a dataset derived from FineWeb and FineWeb2 corpora, covering over 1900 languages and containing frequency counts for 96 trillion characters from 57 TB of text, with per-character statistics and temporal analysis capabilities.
We present FineFreq, a large-scale multilingual character frequency dataset derived from the FineWeb and FineWeb2 corpora, covering over 1900 languages and spanning 2013-2025. The dataset contains frequency counts for 96 trillion characters processed from 57 TB of compressed text. For each language, FineFreq provides per-character statistics with aggregate and year-level frequencies, allowing fine-grained temporal analysis. The dataset preserves naturally occurring multilingual features such as cross-script borrowings, emoji, and acronyms without applying artificial filtering. Each character entry includes Unicode metadata (category, script, block), enabling domain-specific or other downstream filtering and analysis. The full dataset is released in both CSV and Parquet formats, with associated metadata, available on GitHub and HuggingFace. https://github.com/Bin-2/FineFreq