Data-induced multiscale losses and efficient multirate gradient descent schemes
This addresses training inefficiencies in deep learning for datasets with varying scales, though it appears incremental as it builds on existing multiscale algorithms from scientific computing.
The paper tackled the problem of multiscale data distributions in deep learning, revealing multiscale structures in the loss landscape and introducing a novel gradient descent approach that enhances training efficiency, particularly in later stages, by providing a systematic, data-informed strategy.
This paper investigates the impact of multiscale data on machine learning algorithms, particularly in the context of deep learning. A dataset is multiscale if its distribution shows large variations in scale across different directions. This paper reveals multiscale structures in the loss landscape, including its gradients and Hessians inherited from the data. Correspondingly, it introduces a novel gradient descent approach, drawing inspiration from multiscale algorithms used in scientific computing. This approach seeks to transcend empirical learning rate selection, offering a more systematic, data-informed strategy to enhance training efficiency, especially in the later stages.