Learning In Practice: Reasoning About Quantization
This addresses a foundational issue in machine learning theory for researchers and practitioners, though it is incremental as it builds on existing algorithms with quantization-aware modifications.
The paper tackles the mismatch between theoretical machine learning, which assumes real-valued parameters, and practical implementations that use quantized numbers like floating points, by proposing a framework for analyzing learning under arbitrary quantizations and proving convergence for quantization-aware versions of algorithms such as Perceptron and Frank-Wolfe.
There is a mismatch between the standard theoretical analyses of statistical machine learning and how learning is used in practice. The foundational assumption supporting the theory is that we can represent features and models using real-valued parameters. In practice, however, we do not use real numbers at any point during training or deployment. Instead, we rely on discrete and finite quantizations of the reals, typically floating points. In this paper, we propose a framework for reasoning about learning under arbitrary quantizations. Using this formalization, we prove the convergence of quantization-aware versions of the Perceptron and Frank-Wolfe algorithms. Finally, we report the results of an extensive empirical study of the impact of quantization using a broad spectrum of datasets.