Consistency Deep Equilibrium Models
This addresses a computational bottleneck for users of DEQs in deep learning, offering faster inference with maintained performance, though it is incremental as it builds on existing DEQ methods.
The paper tackles the high inference latency of Deep Equilibrium Models (DEQs) by introducing Consistency Deep Equilibrium Models (C-DEQs), which use consistency distillation to accelerate inference, achieving 2-20x accuracy improvements over implicit DEQs under the same few-step inference budget.
Deep Equilibrium Models (DEQs) have emerged as a powerful paradigm in deep learning, offering the ability to model infinite-depth networks with constant memory usage. However, DEQs incur significant inference latency due to the iterative nature of fixed-point solvers. In this work, we introduce the Consistency Deep Equilibrium Model (C-DEQ), a novel framework that leverages consistency distillation to accelerate DEQ inference. We cast the DEQ iterative inference process as evolution along a fixed ODE trajectory toward the equilibrium. Along this trajectory, we train C-DEQs to consistently map intermediate states directly to the fixed point, enabling few-step inference while preserving the performance of the teacher DEQ. At the same time, it facilitates multi-step evaluation to flexibly trade computation for performance gains. Extensive experiments across various domain tasks demonstrate that C-DEQs achieves consistent 2-20$\times$ accuracy improvements over implicit DEQs under the same few-step inference budget.