Normalized Space Alignment: A Versatile Metric for Representation Analysis
This provides a new metric for researchers and practitioners in machine learning to analyze and align neural network representations, though it appears incremental as it builds on existing manifold analysis techniques.
The paper tackles the problem of comparing and aligning neural network representations by introducing Normalized Space Alignment (NSA), a manifold analysis technique that serves as both an analytical tool and differentiable loss function, demonstrating its versatility in representation space analysis, structure preservation, and robustness analysis.
We introduce a manifold analysis technique for neural network representations. Normalized Space Alignment (NSA) compares pairwise distances between two point clouds derived from the same source and having the same size, while potentially possessing differing dimensionalities. NSA can act as both an analytical tool and a differentiable loss function, providing a robust means of comparing and aligning representations across different layers and models. It satisfies the criteria necessary for both a similarity metric and a neural network loss function. We showcase NSA's versatility by illustrating its utility as a representation space analysis metric, a structure-preserving loss function, and a robustness analysis tool. NSA is not only computationally efficient but it can also approximate the global structural discrepancy during mini-batching, facilitating its use in a wide variety of neural network training paradigms.