Hypencoder: Hypernetworks for Information Retrieval
This addresses the problem of limited expressiveness in relevance scoring for information retrieval systems, offering a new paradigm that could enhance search accuracy and efficiency, though it appears incremental in its application to existing retrieval frameworks.
The authors tackled the limitation of vector inner products in information retrieval by introducing Hypencoders, which use hypernetworks to generate query-specific neural networks for scoring relevance, resulting in significant performance improvements over dense retrieval models and even surpassing larger reranking models on in-domain tasks, with a 60-millisecond retrieval time on an 8.8M document corpus.
Existing information retrieval systems are largely constrained by their reliance on vector inner products to assess query-document relevance, which naturally limits the expressiveness of the relevance score they can produce. We propose a new paradigm; instead of representing a query as a vector, we use a small neural network that acts as a learned query-specific relevance function. This small neural network takes a document representation as input (in this work we use a single vector) and produces a scalar relevance score. To produce the small neural network we use a hypernetwork, a network that produces the weights of other networks, as our query encoder. We name this category of encoder models Hypencoders. Experiments on in-domain search tasks show that Hypencoders significantly outperform strong dense retrieval models and even surpass reranking models and retrieval models with an order of magnitude more parameters. To assess the extent of Hypencoders' capabilities, we evaluate on a set of hard retrieval tasks including tip-of-the-tongue and instruction-following retrieval tasks. On harder tasks, we find that the performance gap widens substantially compared to standard retrieval tasks. Furthermore, to demonstrate the practicality of our method, we implement an approximate search algorithm and show that our model is able to retrieve from a corpus of 8.8M documents in under 60 milliseconds.