Sign-Constrained Regularized Loss Minimization
This work addresses the issue of discarding domain knowledge in statistical analysis for non-experts, offering a simple method to integrate sign constraints, though it is incremental as it builds on existing algorithms.
The paper tackles the problem of incorporating domain knowledge, specifically sign constraints, into regularized loss minimization to improve generalization performance, and develops two optimization algorithms (SC-Pega and SC-SDCA) that maintain convergence rates while showing significant improvements in applications like exploiting prior correlations and enhancing SVM-Pairwise.
In practical analysis, domain knowledge about analysis target has often been accumulated, although, typically, such knowledge has been discarded in the statistical analysis stage, and the statistical tool has been applied as a black box. In this paper, we introduce sign constraints that are a handy and simple representation for non-experts in generic learning problems. We have developed two new optimization algorithms for the sign-constrained regularized loss minimization, called the sign-constrained Pegasos (SC-Pega) and the sign-constrained SDCA (SC-SDCA), by simply inserting the sign correction step into the original Pegasos and SDCA, respectively. We present theoretical analyses that guarantee that insertion of the sign correction step does not degrade the convergence rate for both algorithms. Two applications, where the sign-constrained learning is effective, are presented. The one is exploitation of prior information about correlation between explanatory variables and a target variable. The other is introduction of the sign-constrained to SVM-Pairwise method. Experimental results demonstrate significant improvement of generalization performance by introducing sign constraints in both applications.