Geometric Properties of Neural Multivariate Regression
This provides geometric insights for improving generalization in regression tasks in domains like control and robotics, though it is incremental as it extends neural collapse analysis from classification to regression.
The paper tackled the problem of neural collapse degrading performance in neural multivariate regression, finding that collapsed models have lower intrinsic dimension in features than targets, leading to over-compression and poor generalization, with performance depending on data quantity and noise levels.
Neural multivariate regression underpins a wide range of domains such as control, robotics, and finance, yet the geometry of its learned representations remains poorly characterized. While neural collapse has been shown to benefit generalization in classification, we find that analogous collapse in regression consistently degrades performance. To explain this contrast, we analyze models through the lens of intrinsic dimension. Across control tasks and synthetic datasets, we estimate the intrinsic dimension of last-layer features (ID_H) and compare it with that of the regression targets (ID_Y). Collapsed models exhibit ID_H < ID_Y, leading to over-compression and poor generalization, whereas non-collapsed models typically maintain ID_H > ID_Y. For the non-collapsed models, performance with respect to ID_H depends on the data quantity and noise levels. From these observations, we identify two regimes (over-compressed and under-compressed) that determine when expanding or reducing feature dimensionality improves performance. Our results provide new geometric insights into neural regression and suggest practical strategies for enhancing generalization.