Entropy Regularizing Activation: Boosting Continuous Control, Large Language Models, and Image Classification with Activation as Entropy Constraints
This work addresses performance enhancement for machine learning models in areas like LLMs, reinforcement learning, and image classification, presenting a novel paradigm rather than an incremental improvement.
The paper tackles the problem of improving model performance across diverse domains by constraining sampling entropy with specially designed activations, resulting in significant gains such as a 37.4% boost in AIME 2025 score for an LLM, over 30% improvement in continuous control, and a 0.69% increase in ImageNet accuracy.
We propose ERA, a new paradigm that constrains the sampling entropy above given thresholds by applying specially designed activations to the outputs of models. Our approach demonstrates broad effectiveness across different domains: 1) for large language models(LLMs), boosting the AIME 2025 score for Qwen2.5-Math-7B by 37.4%; 2) for continuous control reinforcement learning agents, improving performance by more than 30% over strong baselines such as SAC on the challenging HumanoidBench; 3) for image classification, enhancing ImageNet top-1 accuracy by 0.69% for ResNet-50. These gains are achieved with a computational overhead of less than 7%. Our work validates output activation as a powerful tool for entropy control, opening a new direction for designing simpler and more robust algorithms.