Exploring Data Geometry for Continual Learning
This addresses the challenge of catastrophic forgetting in continual learning for applications with non-Euclidean data, representing an incremental improvement by incorporating geometric structures.
The paper tackles the problem of continual learning for non-stationary data streams by exploring non-Euclidean data geometry, proposing a method that dynamically expands mixed curvature spaces and uses regularization losses to prevent forgetting, achieving better performance than Euclidean baselines.
Continual learning aims to efficiently learn from a non-stationary stream of data while avoiding forgetting the knowledge of old data. In many practical applications, data complies with non-Euclidean geometry. As such, the commonly used Euclidean space cannot gracefully capture non-Euclidean geometric structures of data, leading to inferior results. In this paper, we study continual learning from a novel perspective by exploring data geometry for the non-stationary stream of data. Our method dynamically expands the geometry of the underlying space to match growing geometric structures induced by new data, and prevents forgetting by keeping geometric structures of old data into account. In doing so, making use of the mixed curvature space, we propose an incremental search scheme, through which the growing geometric structures are encoded. Then, we introduce an angular-regularization loss and a neighbor-robustness loss to train the model, capable of penalizing the change of global geometric structures and local geometric structures. Experiments show that our method achieves better performance than baseline methods designed in Euclidean space.