On the Optimization of Margin Distribution
This work addresses a theoretical gap in machine learning for researchers, though it is incremental as it builds on existing margin distribution studies.
The paper tackles the lack of theoretical understanding in optimizing margin distribution by providing a new generalization error bound based on average margin and semi-variance, and proposes MSVMAv, an efficient approach that shows superior performance in experiments.
Margin has played an important role on the design and analysis of learning algorithms during the past years, mostly working with the maximization of the minimum margin. Recent years have witnessed the increasing empirical studies on the optimization of margin distribution according to different statistics such as medium margin, average margin, margin variance, etc., whereas there is a relative paucity of theoretical understanding. In this work, we take one step on this direction by providing a new generalization error bound, which is heavily relevant to margin distribution by incorporating ingredients such as average margin and semi-variance, a new margin statistics for the characterization of margin distribution. Inspired by the theoretical findings, we propose the MSVMAv, an efficient approach to achieve better performance by optimizing margin distribution in terms of its empirical average margin and semi-variance. We finally conduct extensive experiments to show the superiority of the proposed MSVMAv approach.