Fine-tune your Classifier: Finding Correlations With Temperature
This work addresses a hyperparameter tuning challenge for machine learning practitioners, but it is incremental as it builds on existing methods for temperature analysis.
The authors tackled the problem of selecting the optimal temperature hyperparameter in neural network classification tasks by analyzing dataset statistics to predict a default value, finding promising correlations across over a hundred dataset and feature extractor combinations.
Temperature is a widely used hyperparameter in various tasks involving neural networks, such as classification or metric learning, whose choice can have a direct impact on the model performance. Most of existing works select its value using hyperparameter optimization methods requiring several runs to find the optimal value. We propose to analyze the impact of temperature on classification tasks by describing a dataset as a set of statistics computed on representations on which we can build a heuristic giving us a default value of temperature. We study the correlation between these extracted statistics and the observed optimal temperatures. This preliminary study on more than a hundred combinations of different datasets and features extractors highlights promising results towards the construction of a general heuristic for temperature.