ScaleAcross: Designing Multi-Data-Center Infrastructure for Geo-Distributed AI Training
For researchers and engineers designing multi-data-center infrastructure for geo-distributed AI training, this work provides a reproducible emulation framework and insights into communication behavior, though it is incremental in nature.
The paper investigates EVPN-VXLAN as an infrastructure foundation for geo-distributed AI training and presents a scalable emulation framework that incorporates ECMP, BFD, and a queue-pair-aware traffic distribution mechanism. Results provide insights into traffic distribution and resilience under distributed training workloads with AllReduce and Parameter Server patterns.
The rapid growth of AI models and increasing data sovereignty requirements are driving the transition toward geo-distributed AI training across multiple data centers. Such deployments introduce system-level challenges arising from synchronization-intensive communication, cross-site data exchange, and wide-area latency constraints. This paper investigates EVPN--VXLAN as an infrastructure foundation for geo-distributed AI training environments and presents a scalable emulation framework for systematically studying distributed AI workloads under realistic wide-area conditions. The proposed framework combines VXLAN overlays with EVPN-based inter-data-center connectivity and is implemented using ContainerLab and FRRouting (FRR). The framework further incorporates Equal-Cost Multi-Path (ECMP) routing, Bidirectional Forwarding Detection (BFD), and a queue-pair-aware traffic distribution mechanism designed to improve communication behavior for synchronization-intensive AI workloads while preserving compatibility with commodity infrastructure. Using realistic WAN emulation, we characterize communication and system behavior under distributed training workloads employing AllReduce and Parameter Server communication patterns. Results provide insights into traffic distribution, resilience, and infrastructure behavior in geo-distributed AI environments, highlighting the potential of reproducible multi-data-center infrastructure frameworks for scalable distributed AI training.