Incrementally-Computable Neural Networks: Efficient Inference for Dynamic Inputs
This addresses the challenge of expensive re-computation for dynamic inputs in AI applications, offering a domain-specific improvement for efficient inference.
The paper tackles the problem of efficiently processing dynamic inputs in deep learning, such as real-time updates for AI writing assistants, by introducing an incremental inference algorithm that uses vector quantization to discretize intermediate values, achieving comparable accuracy on document classification with 12.1X fewer operations for sequences of atomic edits.
Deep learning often faces the challenge of efficiently processing dynamic inputs, such as sensor data or user inputs. For example, an AI writing assistant is required to update its suggestions in real time as a document is edited. Re-running the model each time is expensive, even with compression techniques like knowledge distillation, pruning, or quantization. Instead, we take an incremental computing approach, looking to reuse calculations as the inputs change. However, the dense connectivity of conventional architectures poses a major obstacle to incremental computation, as even minor input changes cascade through the network and restrict information reuse. To address this, we use vector quantization to discretize intermediate values in the network, which filters out noisy and unnecessary modifications to hidden neurons, facilitating the reuse of their values. We apply this approach to the transformers architecture, creating an efficient incremental inference algorithm with complexity proportional to the fraction of the modified inputs. Our experiments with adapting the OPT-125M pre-trained language model demonstrate comparable accuracy on document classification while requiring 12.1X (median) fewer operations for processing sequences of atomic edits.