GPUs, CPUs, and... NICs: Rethinking the Network's Role in Serving Complex AI Pipelines
This addresses inefficiencies in AI inference platforms for developers and operators, but it is incremental as it builds on existing offloading concepts.
The paper tackles the problem of network delays and high resource overheads in complex AI pipelines by proposing to offload data processing tasks onto SmartNICs, aiming to optimize distributed serving.
The increasing prominence of AI necessitates the deployment of inference platforms for efficient and effective management of AI pipelines and compute resources. As these pipelines grow in complexity, the demand for distributed serving rises and introduces much-dreaded network delays. In this paper, we investigate how the network can instead be a boon to the excessively high resource overheads of AI pipelines. To alleviate these overheads, we discuss how resource-intensive data processing tasks -- a key facet of growing AI pipeline complexity -- are well-matched for the computational characteristics of packet processing pipelines and how they can be offloaded onto SmartNICs. We explore the challenges and opportunities of offloading, and propose a research agenda for integrating network hardware into AI pipelines, unlocking new opportunities for optimization.