Sparse by Command: Task-Conditional Compute Skipping for Multi-Task Inference Accelerators
For multi-task inference accelerators, this provides a practical method to reduce computation and energy without altering model architecture, achieving significant efficiency gains with minimal overhead.
This work introduces a HW/SW co-designed approach for multi-task inference that uses a lightweight gating network to predict per-tile binary execution masks conditioned on the task command, enabling task-dependent compute skipping. On a visuomotor driving task, it reduces FLOPs by 66-76%, decreases latency by 51-59% (2.1-2.4x speedup), and cuts energy per inference from 263 to 108-128mJ while maintaining driving quality.
Multi-task inference models share a single backbone across diverse tasks, yet execute identical computation regardless of which task is active - wasting energy and cycles on task-irrelevant operations. We observe that the task command, typically available before inference begins, provides a free signal that can be exploited to skip unnecessary computation at the hardware level. We present a HW/SW co-designed approach in which a lightweight gating network, trained jointly with the backbone, predicts per-tile binary execution masks conditioned on the task input. Each tile corresponds to a fixed group of output channels (the native scheduling granularity of the accelerator), enabling masked tiles to be skipped with zero overhead. This yields a task-dependent reduction in compute, where each command activates only the subset of the network it requires, without changes to the model architecture or inference pipeline. We co-design the full system stack: a command-conditioned training procedure that learns hardware-aligned tile masks under a sparsity objective; an instruction set architecture whose instructions carry per-tile bitmask fields, allowing the hardware to skip masked tiles without software intervention; and a tiled inference accelerator with configurable parallelism, double-buffered memory, and INT8 datapath that natively supports sparse tile execution. We prototype on an AMD/Xilinx Alveo U50 FPGA and evaluate on a closed-loop visuomotor driving task in CARLA autonomous driving simulator. Task-conditional sparsity reduces FLOPs by 66-76% while maintaining driving quality. On-device latency decreases by 51-59%, from 9.12 ms to 3.74-4.44 ms (2.1-2.4x speedup), with energy per inference dropping from 263 to 108-128mJ.