Nonextensive information theoretical machine
This work proposes a novel theoretical framework for machine learning models, but it appears incremental as it builds on existing information theory concepts without clear broad impact.
The paper introduces the nonextensive information theoretical machine (NITM), a discriminative model that unifies margin-based loss functions using Tsallis entropy and generalizes Gaussian prior regularization to Student-t prior, with performance demonstrated on standard datasets.
In this paper, we propose a new discriminative model named \emph{nonextensive information theoretical machine (NITM)} based on nonextensive generalization of Shannon information theory. In NITM, weight parameters are treated as random variables. Tsallis divergence is used to regularize the distribution of weight parameters and maximum unnormalized Tsallis entropy distribution is used to evaluate fitting effect. On the one hand, it is showed that some well-known margin-based loss functions such as $\ell_{0/1}$ loss, hinge loss, squared hinge loss and exponential loss can be unified by unnormalized Tsallis entropy. On the other hand, Gaussian prior regularization is generalized to Student-t prior regularization with similar computational complexity. The model can be solved efficiently by gradient-based convex optimization and its performance is illustrated on standard datasets.