Penalizing Localized Dirichlet Energies in Low Rank Tensor Products
This work addresses overfitting in regression tasks for researchers and practitioners using tensor-product models, offering an incremental improvement in regularization techniques.
The paper tackles the problem of overfitting in low-rank tensor-product B-spline models by proposing a novel regularization strategy based on local Dirichlet energies, showing that these models outperform neural networks in overfitting regimes for most datasets and maintain competitive performance otherwise.
We study low-rank tensor-product B-spline (TPBS) models for regression tasks and investigate Dirichlet energy as a measure of smoothness. We show that TPBS models admit a closed-form expression for the Dirichlet energy, and reveal scenarios where perfect interpolation is possible with exponentially small Dirichlet energy. This renders global Dirichlet energy-based regularization ineffective. To address this limitation, we propose a novel regularization strategy based on local Dirichlet energies defined on small hypercubes centered at the training points. Leveraging pretrained TPBS models, we also introduce two estimators for inference from incomplete samples. Comparative experiments with neural networks demonstrate that TPBS models outperform neural networks in the overfitting regime for most datasets, and maintain competitive performance otherwise. Overall, TPBS models exhibit greater robustness to overfitting and consistently benefit from regularization, while neural networks are more sensitive to overfitting and less effective in leveraging regularization.