On Equivariant Model Selection through the Lens of Uncertainty
This work addresses model selection for researchers using equivariant models, but it is incremental as it compares existing uncertainty methods without introducing new ones.
The paper tackled the problem of selecting among pretrained equivariant models with varying symmetry biases, finding that uncertainty metrics generally align with predictive performance, but Bayesian model evidence does so consistently due to a mismatch in complexity notions.
Equivariant models leverage prior knowledge on symmetries to improve predictive performance, but misspecified architectural constraints can harm it instead. While work has explored learning or relaxing constraints, selecting among pretrained models with varying symmetry biases remains challenging. We examine this model selection task from an uncertainty-aware perspective, comparing frequentist (via Conformal Prediction), Bayesian (via the marginal likelihood), and calibration-based measures to naive error-based evaluation. We find that uncertainty metrics generally align with predictive performance, but Bayesian model evidence does so inconsistently. We attribute this to a mismatch in Bayesian and geometric notions of model complexity for the employed last-layer Laplace approximation, and discuss possible remedies. Our findings point towards the potential of uncertainty in guiding symmetry-aware model selection.