From Low Intrinsic Dimensionality to Non-Vacuous Generalization Bounds in Deep Multi-Task Learning
This provides theoretical guarantees for generalization in deep multi-task learning, which is incremental but addresses a key bottleneck in understanding overparametrized models.
The paper tackles the problem of overfitting in deep multi-task learning by showing that models operate in low intrinsic dimensionality, enabling high-accuracy solutions with fewer parameters than single-task learning, and it derives the first non-vacuous generalization bounds for such networks.
Deep learning methods are known to generalize well from training to future data, even in an overparametrized regime, where they could easily overfit. One explanation for this phenomenon is that even when their *ambient dimensionality*, (i.e. the number of parameters) is large, the models' *intrinsic dimensionality* is small; specifically, their learning takes place in a small subspace of all possible weight configurations. In this work, we confirm this phenomenon in the setting of *deep multi-task learning*. We introduce a method to parametrize multi-task network directly in the low-dimensional space, facilitated by the use of *random expansions* techniques. We then show that high-accuracy multi-task solutions can be found with much smaller intrinsic dimensionality (fewer free parameters) than what single-task learning requires. Subsequently, we show that the low-dimensional representations in combination with *weight compression* and *PAC-Bayesian* reasoning lead to the *first non-vacuous generalization bounds* for deep multi-task networks.