Entropy-UID: A Method for Optimizing Information Density
This addresses the problem of inefficient text generation for users of autoregressive language models, though it appears incremental as it refines existing token selection strategies.
The paper tackled the problem of optimizing information flow in language generation models by proposing Entropy-UID, a token selection method that balances entropy and Uniform Information Density principles. The result showed that Entropy-UID achieved lower surprisal and entropy variance compared to standard GPT-2 and alternative heuristics on datasets like WikiText-2, OpenWebText, and WMT.
Balanced and efficient information flow is essential for optimizing language generation models. In this work, we propose Entropy-UID, a new token selection method that balances entropy and Uniform Information Density (UID) principles for enhanced efficiency of text generation. Our approach adaptively adjusts token selection by jointly minimizing entropy and surprisal, promoting more even information distribution across generated sequences. Theoretical validation demonstrates that Entropy-UID optimally reduces information spikes while maintaining fluency and coherence. The method has been evulated using information-theoretic metrics on multiple benchmark datasets, including WikiText-2, OpenWebText, and WMT. Experimental results show that Entropy-UID achieves lower surprisal and entropy variance compared to standard GPT-2 and alternative heuristics, leading to more balanced and human-like text generation. Our findings point towards the potential of leveraging information-theoretic constraints to refine token selection strategies in autoregressive language models.