Rediscovering Hashed Random Projections for Efficient Quantization of Contextualized Sentence Embeddings
This addresses storage and bandwidth challenges for deploying models on edge devices, but it is incremental as it builds on existing quantization and projection techniques.
The paper tackles the problem of reducing storage and bandwidth for pre-computed sentence embeddings on edge devices by proposing a method using random hyperplane projections and quantization, achieving up to 98.96% size reduction while retaining 94%–99% of performance on classification tasks.
Training and inference on edge devices often requires an efficient setup due to computational limitations. While pre-computing data representations and caching them on a server can mitigate extensive edge device computation, this leads to two challenges. First, the amount of storage required on the server that scales linearly with the number of instances. Second, the bandwidth required to send extensively large amounts of data to an edge device. To reduce the memory footprint of pre-computed data representations, we propose a simple, yet effective approach that uses randomly initialized hyperplane projections. To further reduce their size by up to 98.96%, we quantize the resulting floating-point representations into binary vectors. Despite the greatly reduced size, we show that the embeddings remain effective for training models across various English and German sentence classification tasks that retain 94%--99% of their floating-point.