Dual Codebook VQ: Enhanced Image Reconstruction with Reduced Codebook Size
This work addresses the challenge of high-fidelity image reconstruction with reduced computational requirements for applications in computer vision, though it appears incremental as it builds on existing VQ techniques.
The paper tackles the problem of limited codebook utilization in Vector Quantization (VQ) for image modeling by introducing a Dual Codebook mechanism that partitions representations into global and local components, achieving state-of-the-art reconstruction quality with a compact codebook size of 512—half the size of previous methods—and significant FID improvements across diverse image domains.
Vector Quantization (VQ) techniques face significant challenges in codebook utilization, limiting reconstruction fidelity in image modeling. We introduce a Dual Codebook mechanism that effectively addresses this limitation by partitioning the representation into complementary global and local components. The global codebook employs a lightweight transformer for concurrent updates of all code vectors, while the local codebook maintains precise feature representation through deterministic selection. This complementary approach is trained from scratch without requiring pre-trained knowledge. Experimental evaluation across multiple standard benchmark datasets demonstrates state-of-the-art reconstruction quality while using a compact codebook of size 512 - half the size of previous methods that require pre-training. Our approach achieves significant FID improvements across diverse image domains, particularly excelling in scene and face reconstruction tasks. These results establish Dual Codebook VQ as an efficient paradigm for high-fidelity image reconstruction with significantly reduced computational requirements.