Topological Obstructions and How to Avoid Them
This addresses challenges in incorporating geometric inductive biases for interpretability and generalization in machine learning models, though it appears incremental as it builds on existing flow-based methods to handle specific topological issues.
The paper tackles the problem of training encoders with geometric latent spaces, which face topological obstructions like singularities and incorrect winding numbers, leading to local optima. It proposes a flow-based model that maps data to multimodal distributions over geometric spaces, resulting in improved training stability and a higher chance of converging to a homeomorphic encoder in empirical evaluations on two domains.
Incorporating geometric inductive biases into models can aid interpretability and generalization, but encoding to a specific geometric structure can be challenging due to the imposed topological constraints. In this paper, we theoretically and empirically characterize obstructions to training encoders with geometric latent spaces. We show that local optima can arise due to singularities (e.g. self-intersection) or due to an incorrect degree or winding number. We then discuss how normalizing flows can potentially circumvent these obstructions by defining multimodal variational distributions. Inspired by this observation, we propose a new flow-based model that maps data points to multimodal distributions over geometric spaces and empirically evaluate our model on 2 domains. We observe improved stability during training and a higher chance of converging to a homeomorphic encoder.