Uncertainty in Semantic Language Modeling with PIXELS
This addresses the uncertainty quantification problem for researchers using pixel-based language models, but it appears incremental as it applies existing methods to analyze a known bottleneck.
The paper tackled uncertainty quantification in pixel-based language models across 18 languages and 7 scripts, finding that these models underestimate uncertainty in patch reconstruction and that uncertainty varies by script, with Latin languages showing lower uncertainty.
Pixel-based language models aim to solve the vocabulary bottleneck problem in language modeling, but the challenge of uncertainty quantification remains open. The novelty of this work consists of analysing uncertainty and confidence in pixel-based language models across 18 languages and 7 scripts, all part of 3 semantically challenging tasks. This is achieved through several methods such as Monte Carlo Dropout, Transformer Attention, and Ensemble Learning. The results suggest that pixel-based models underestimate uncertainty when reconstructing patches. The uncertainty is also influenced by the script, with Latin languages displaying lower uncertainty. The findings on ensemble learning show better performance when applying hyperparameter tuning during the named entity recognition and question-answering tasks across 16 languages.