Rectified Lagrangian for Out-of-Distribution Detection in Modern Hopfield Networks
This addresses a specific bottleneck in Hopfield networks for AI applications, offering an incremental improvement in OOD detection.
The paper tackles the problem of out-of-distribution (OOD) detection in modern Hopfield networks by proposing a rectified Lagrangian that explicitly incorporates an OOD attractor, enabling OOD identification through trivial point attractors and optimized interaction matrices. It demonstrates effectiveness compared to energy-based methods across nine image datasets.
Modern Hopfield networks (MHNs) have recently gained significant attention in the field of artificial intelligence because they can store and retrieve a large set of patterns with an exponentially large memory capacity. A MHN is generally a dynamical system defined with Lagrangians of memory and feature neurons, where memories associated with in-distribution (ID) samples are represented by attractors in the feature space. One major problem in existing MHNs lies in managing out-of-distribution (OOD) samples because it was originally assumed that all samples are ID samples. To address this, we propose the rectified Lagrangian (RegLag), a new Lagrangian for memory neurons that explicitly incorporates an attractor for OOD samples in the dynamical system of MHNs. RecLag creates a trivial point attractor for any interaction matrix, enabling OOD detection by identifying samples that fall into this attractor as OOD. The interaction matrix is optimized so that the probability densities can be estimated to identify ID/OOD. We demonstrate the effectiveness of RecLag-based MHNs compared to energy-based OOD detection methods, including those using state-of-the-art Hopfield energies, across nine image datasets.