CQ-CiM: Hardware-Aware Embedding Shaping for Robust CiM-Based Retrieval
This work solves the representation gap for hardware designers and edge device users, enabling robust CiM-based retrieval for RAG, though it appears incremental as it builds on existing data shaping methods by unifying them.
The paper tackles the problem of deploying Retrieval-Augmented Generation (RAG) on edge devices by addressing the incompatibility of high-precision, high-dimension embeddings with Compute-in-Memory (CiM) architectures, introducing CQ-CiM as a unified framework that jointly learns compression and quantization to produce CiM-compatible low-bit embeddings for diverse CiM designs.
Deploying Retrieval-Augmented Generation (RAG) on edge devices is in high demand, but is hindered by the latency of massive data movement and computation on traditional architectures. Compute-in-Memory (CiM) architectures address this bottleneck by performing vector search directly within their crossbar structure. However, CiM's adoption for RAG is limited by a fundamental ``representation gap,'' as high-precision, high-dimension embeddings are incompatible with CiM's low-precision, low-dimension array constraints. This gap is compounded by the diversity of CiM implementations (e.g., SRAM, ReRAM, FeFET), each with unique designs (e.g., 2-bit cells, 512x512 arrays). Consequently, RAG data must be naively reshaped to fit each target implementation. Current data shaping methods handle dimension and precision disjointly, which degrades data fidelity. This not only negates the advantages of CiM for RAG but also confuses hardware designers, making it unclear if a failure is due to the circuit design or the degraded input data. As a result, CiM adoption remains limited. In this paper, we introduce CQ-CiM, a unified, hardware-aware data shaping framework that jointly learns Compression and Quantization to produce CiM-compatible low-bit embeddings for diverse CiM designs. To the best of our knowledge, this is the first work to shape data for comprehensive CiM usage on RAG.