Cascade: A Platform for Delay-Sensitive Edge Intelligence
This addresses latency issues for edge intelligence applications, though it appears incremental as it builds on existing platform optimizations.
The paper tackles the problem of high tail-latency in AI/ML platforms for interactive applications by introducing Cascade, a platform that reduces latency by orders of magnitude while maintaining throughput.
Interactive intelligent computing applications are increasingly prevalent, creating a need for AI/ML platforms optimized to reduce per-event latency while maintaining high throughput and efficient resource management. Yet many intelligent applications run on AI/ML platforms that optimize for high throughput even at the cost of high tail-latency. Cascade is a new AI/ML hosting platform intended to untangle this puzzle. Innovations include a legacy-friendly storage layer that moves data with minimal copying and a "fast path" that collocates data and computation to maximize responsiveness. Our evaluation shows that Cascade reduces latency by orders of magnitude with no loss of throughput.