One-Bit Quantization for Random Features Models
This work provides theoretical insights into neural network compression for resource-constrained devices, though it is incremental as it builds on existing Random Features models.
The paper tackles the problem of high computational and memory demands in neural networks by analyzing one-bit weight quantization in Random Features models, proving asymptotically no loss in generalization error when quantizing all layers except the last and demonstrating significant inference speed-ups on a laptop GPU.
Recent advances in neural networks have led to significant computational and memory demands, spurring interest in one-bit weight compression to enable efficient inference on resource-constrained devices. However, the theoretical underpinnings of such compression remain poorly understood. We address this gap by analyzing one-bit quantization in the Random Features model, a simplified framework that corresponds to neural networks with random representations. We prove that, asymptotically, quantizing weights of all layers except the last incurs no loss in generalization error, compared to the full precision random features model. Our findings offer theoretical insights into neural network compression. We also demonstrate empirically that one-bit quantization leads to significant inference speed ups for the Random Features models even on a laptop GPU, confirming the practical benefits of our work. Additionally, we provide an asymptotically precise characterization of the generalization error for Random Features with an arbitrary number of layers. To the best of our knowledge, our analysis yields more general results than all previous works in the related literature.