A Unified Probabilistic Framework for Dictionary Learning with Parsimonious Activation
This work addresses the issue of inefficient dictionary learning for signal processing and machine learning applications, offering an incremental improvement by incorporating a novel regularization approach to enhance atom sharing across samples.
The paper tackled the problem of redundant and sub-optimal dictionaries in dictionary learning by introducing a parsimony-promoting regularizer based on the row-wise L∞ norm, which reduced the number of activated atoms across the dataset. The result was a 20% reduction in RMSE and the use of fewer than one-tenth of the available dictionary atoms, improving reconstruction quality and sparsity.
Dictionary learning is traditionally formulated as an $L_1$-regularized signal reconstruction problem. While recent developments have incorporated discriminative, hierarchical, or generative structures, most approaches rely on encouraging representation sparsity over individual samples that overlook how atoms are shared across samples, resulting in redundant and sub-optimal dictionaries. We introduce a parsimony promoting regularizer based on the row-wise $L_\infty$ norm of the coefficient matrix. This additional penalty encourages entire rows of the coefficient matrix to vanish, thereby reducing the number of dictionary atoms activated across the dataset. We derive the formulation from a probabilistic model with Beta-Bernoulli priors, which provides a Bayesian interpretation linking the regularization parameters to prior distributions. We further establish theoretical calculation for optimal hyperparameter selection and connect our formulation to both Minimum Description Length, Bayesian model selection and pathlet learning. Extensive experiments on benchmark datasets demonstrate that our method achieves substantially improved reconstruction quality (with a 20\% reduction in RMSE) and enhanced representation sparsity, utilizing fewer than one-tenth of the available dictionary atoms, while empirically validating our theoretical analysis.