Operator-Based Generalization Bound for Deep Learning: Insights on Multi-Task Learning
This work addresses the relatively unexplored area of generalization properties in multi-task learning with deep architectures, offering insights for applications like robust and quantile regression.
The paper tackles the problem of deriving tighter generalization bounds for vector-valued neural networks and deep kernel methods in multi-task learning, achieving improved guarantees through an operator-theoretic framework and sketching techniques.
This paper presents novel generalization bounds for vector-valued neural networks and deep kernel methods, focusing on multi-task learning through an operator-theoretic framework. Our key development lies in strategically combining a Koopman based approach with existing techniques, achieving tighter generalization guarantees compared to traditional norm-based bounds. To mitigate computational challenges associated with Koopman-based methods, we introduce sketching techniques applicable to vector valued neural networks. These techniques yield excess risk bounds under generic Lipschitz losses, providing performance guarantees for applications including robust and multiple quantile regression. Furthermore, we propose a novel deep learning framework, deep vector-valued reproducing kernel Hilbert spaces (vvRKHS), leveraging Perron Frobenius (PF) operators to enhance deep kernel methods. We derive a new Rademacher generalization bound for this framework, explicitly addressing underfitting and overfitting through kernel refinement strategies. This work offers novel insights into the generalization properties of multitask learning with deep learning architectures, an area that has been relatively unexplored until recent developments.