Efficient Multi-stage Inference on Tabular Data
This addresses efficiency issues for ML systems in production, particularly for high-throughput applications with tabular data, though it is incremental as it optimizes existing inference methods.
The paper tackles the bottleneck of real-time inference in ML systems by embedding simplified inference algorithms into product code to reduce network latency and CPU overhead, achieving a 1.3x reduction in inference latency, 30% lower CPU usage, and 50% less network communication for a commercial platform handling millions of decisions per second.
Many ML applications and products train on medium amounts of input data but get bottlenecked in real-time inference. When implementing ML systems, conventional wisdom favors segregating ML code into services queried by product code via Remote Procedure Call (RPC) APIs. This approach clarifies the overall software architecture and simplifies product code by abstracting away ML internals. However, the separation adds network latency and entails additional CPU overhead. Hence, we simplify inference algorithms and embed them into the product code to reduce network communication. For public datasets and a high-performance real-time platform that deals with tabular data, we show that over half of the inputs are often amenable to such optimization, while the remainder can be handled by the original model. By applying our optimization with AutoML to both training and inference, we reduce inference latency by 1.3x, CPU resources by 30%, and network communication between application front-end and ML back-end by about 50% for a commercial end-to-end ML platform that serves millions of real-time decisions per second.