Contextual Temperature for Language Modeling
This work provides a more adaptive and effective temperature scaling method for language models, which could benefit researchers and practitioners working on improving language model performance and uncertainty control.
This paper addresses the limitation of fixed or manually scheduled temperature scaling in language models by proposing a contextual temperature approach that learns an optimal temperature trajectory for each vocabulary item based on its context. This method significantly improved state-of-the-art language models, achieving perplexity scores of 55.31 on Penn Treebank and 62.89 on WikiText-2.
Temperature scaling has been widely used as an effective approach to control the smoothness of a distribution, which helps the model performance in various tasks. Current practices to apply temperature scaling assume either a fixed, or a manually-crafted dynamically changing schedule. However, our studies indicate that the individual optimal trajectory for each class can change with the context. To this end, we propose contextual temperature, a generalized approach that learns an optimal temperature trajectory for each vocabulary over the context. Experimental results confirm that the proposed method significantly improves state-of-the-art language models, achieving a perplexity of 55.31 and 62.89 on the test set of Penn Treebank and WikiText-2, respectively. In-depth analyses show that the behaviour of the learned temperature schedules varies dramatically by vocabulary, and that the optimal schedules help in controlling the uncertainties. These evidences further justify the need for the proposed method and its advantages over fixed temperature schedules.