From seeing to remembering: Images with harder-to-reconstruct representations leave stronger memory traces
This work addresses a foundational question in cognitive science about the interface between perception and memory, offering a novel signal for memory encoding, though it is incremental in building on the level-of-processing theory.
The researchers tackled the problem of how perception influences memory by proposing a sparse coding model that compresses image feature embeddings, and found that reconstruction error predicts memory accuracy and retrieval latencies, explaining all variance from vision-only models in the latter case.
Much of what we remember is not due to intentional selection, but simply a by-product of perceiving. This raises a foundational question about the architecture of the mind: How does perception interface with and influence memory? Here, inspired by a classic proposal relating perceptual processing to memory durability, the level-of-processing theory, we present a sparse coding model for compressing feature embeddings of images, and show that the reconstruction residuals from this model predict how well images are encoded into memory. In an open memorability dataset of scene images, we show that reconstruction error not only explains memory accuracy but also response latencies during retrieval, subsuming, in the latter case, all of the variance explained by powerful vision-only models. We also confirm a prediction of this account with 'model-driven psychophysics'. This work establishes reconstruction error as a novel signal interfacing perception and memory, possibly through adaptive modulation of perceptual processing.