Neural Networks Learn Distance Metrics
This work addresses a foundational issue in neural network design, potentially influencing how models are built and interpreted, though it appears incremental in exploring representation types.
The paper tackled the problem of whether neural networks naturally favor distance-based representations over intensity-based ones, finding that the underlying representation type affects model performance, as demonstrated through experiments with six MNIST architectural variants.
Neural networks may naturally favor distance-based representations, where smaller activations indicate closer proximity to learned prototypes. This contrasts with intensity-based approaches, which rely on activation magnitudes. To test this hypothesis, we conducted experiments with six MNIST architectural variants constrained to learn either distance or intensity representations. Our results reveal that the underlying representation affects model performance. We develop a novel geometric framework that explains these findings and introduce OffsetL2, a new architecture based on Mahalanobis distance equations, to further validate this framework. This work highlights the importance of considering distance-based learning in neural network design.