Maximum Regularized Likelihood Estimators: A General Prediction Theory and Applications
This work addresses a foundational issue in statistical learning by offering a general prediction theory applicable to a wide range of models, such as tensor regression and graphical models, which is incremental as it extends existing frameworks.
The paper tackles the problem of providing prediction accuracy guarantees for maximum regularized likelihood estimators (MRLEs) in high-dimensional statistics, showing that MRLEs are broadly consistent in prediction under general conditions without requiring restrictive assumptions like restricted eigenvalues.
Maximum regularized likelihood estimators (MRLEs) are arguably the most established class of estimators in high-dimensional statistics. In this paper, we derive guarantees for MRLEs in Kullback-Leibler divergence, a general measure of prediction accuracy. We assume only that the densities have a convex parametrization and that the regularization is definite and positive homogenous. The results thus apply to a very large variety of models and estimators, such as tensor regression and graphical models with convex and non-convex regularized methods. A main conclusion is that MRLEs are broadly consistent in prediction - regardless of whether restricted eigenvalues or similar conditions hold.