Landscaper: Understanding Loss Landscapes Through Multi-Dimensional Topological Analysis
This work addresses the challenge of analyzing complex loss landscapes for researchers in machine learning, particularly in data-scarce scientific domains, though it is incremental as it builds on existing topological methods.
The paper tackled the problem of understanding neural network optimization and generalization by developing Landscaper, a tool for multi-dimensional loss landscape analysis, which revealed geometric structures like basin hierarchy and introduced the SMAD metric to capture training transitions and out-of-distribution generalization insights.
Loss landscapes are a powerful tool for understanding neural network optimization and generalization, yet traditional low-dimensional analyses often miss complex topological features. We present Landscaper, an open-source Python package for arbitrary-dimensional loss landscape analysis. Landscaper combines Hessian-based subspace construction with topological data analysis to reveal geometric structures such as basin hierarchy and connectivity. A key component is the Saddle-Minimum Average Distance (SMAD) for quantifying landscape smoothness. We demonstrate Landscaper's effectiveness across various architectures and tasks, including those involving pre-trained language models, showing that SMAD captures training transitions, such as landscape simplification, that conventional metrics miss. We also illustrate Landscaper's performance in challenging chemical property prediction tasks, where SMAD can serve as a metric for out-of-distribution generalization, offering valuable insights for model diagnostics and architecture design in data-scarce scientific machine learning scenarios.