ZOQO: Zero-Order Quantized Optimization
This addresses resource constraints in deep learning, offering a practical solution for low-resource environments, though it appears incremental as it builds on existing zero-order and quantization techniques.
The paper tackles the challenge of high computational and memory demands in deep learning by introducing ZOQO, a zero-order quantized optimization method for training models with quantized parameters and operations, achieving competitive performance compared to full-precision methods in fine-tuning large language models and black-box adversarial attacks.
The increasing computational and memory demands in deep learning present significant challenges, especially in resource-constrained environments. We introduce a zero-order quantized optimization (ZOQO) method designed for training models with quantized parameters and operations. Our approach leverages zero-order approximations of the gradient sign and adapts the learning process to maintain the parameters' quantization without the need for full-precision gradient calculations. We demonstrate the effectiveness of ZOQO through experiments in fine-tuning of large language models and black-box adversarial attacks. Despite the limitations of zero-order and quantized operations training, our method achieves competitive performance compared to full-precision methods, highlighting its potential for low-resource environments.