AI Greenferencing: Routing AI Inferencing to Green Modular Data Centers with Heron
This addresses the challenge of reducing AI's environmental impact and costs by leveraging green energy, though it is an incremental improvement in workload routing for specific domains.
The paper tackles the problem of AI inferencing's high power demand by routing workloads to modular data centers at wind farms, using a software router called Heron to improve aggregate goodput by up to 80% compared to state-of-the-art methods.
AI power demand is growing unprecedentedly thanks to the high power density of AI compute and the emerging inferencing workload. On the supply side, abundant wind power is waiting for grid access in interconnection queues. In this light, this paper argues bringing AI workload to modular compute clusters co-located in wind farms. Our deployment right-sizing strategy makes it economically viable to deploy more than 6 million high-end GPUs today that could consume cheap, green power at its source. We built Heron, a cross-site software router, that could efficiently leverage the complementarity of power generation across wind farms by routing AI inferencing workload around power drops. Using 1-week ofcoding and conversation production traces from Azure and (real) variable wind power traces, we show how Heron improves aggregate goodput of AI compute by up to 80% compared to the state-of-the-art.