Neural Networks Use Distance Metrics
This challenges the intensity-based interpretation of neural networks, offering new insights for researchers in machine learning.
The paper tackled the problem of understanding neural network representations by showing that networks with ReLU and Absolute Value activations learn distance-based representations, finding high sensitivity to distance perturbations but robustness to intensity changes.
We present empirical evidence that neural networks with ReLU and Absolute Value activations learn distance-based representations. We independently manipulate both distance and intensity properties of internal activations in trained models, finding that both architectures are highly sensitive to small distance-based perturbations while maintaining robust performance under large intensity-based perturbations. These findings challenge the prevailing intensity-based interpretation of neural network activations and offer new insights into their learning and decision-making processes.