Straightening Out the Straight-Through Estimator: Overcoming Optimization Challenges in Vector Quantized Networks
This addresses training instability problems for researchers and practitioners using vector quantization in neural networks, representing an incremental improvement to existing methods.
This work tackles optimization challenges in vector quantized networks using straight-through estimation, identifying distribution discrepancies and gradient issues as primary causes of instability. The proposed methods including affine re-parameterization, alternating optimization, and improved commitment loss demonstrate effectiveness across AlexNet, ResNet, and ViT architectures on image classification and generative modeling tasks.
This work examines the challenges of training neural networks using vector quantization using straight-through estimation. We find that a primary cause of training instability is the discrepancy between the model embedding and the code-vector distribution. We identify the factors that contribute to this issue, including the codebook gradient sparsity and the asymmetric nature of the commitment loss, which leads to misaligned code-vector assignments. We propose to address this issue via affine re-parameterization of the code vectors. Additionally, we introduce an alternating optimization to reduce the gradient error introduced by the straight-through estimation. Moreover, we propose an improvement to the commitment loss to ensure better alignment between the codebook representation and the model embedding. These optimization methods improve the mathematical approximation of the straight-through estimation and, ultimately, the model performance. We demonstrate the effectiveness of our methods on several common model architectures, such as AlexNet, ResNet, and ViT, across various tasks, including image classification and generative modeling.