Pathway: a fast and flexible unified stream data processing framework for analytical and Machine Learning applications
This framework addresses the need for fast and flexible stream processing in analytical and machine learning applications, particularly for IoT and enterprise systems, offering incremental improvements over existing industry solutions.
The authors introduced Pathway, a unified data processing framework for bounded and unbounded data streams, designed to handle IoT and enterprise data with rapid reaction and advanced computation like machine learning. Benchmarking showed it surpasses state-of-the-art industry frameworks in both batch and streaming contexts, including handling streaming iterative graph algorithms like PageRank.
We present Pathway, a new unified data processing framework that can run workloads on both bounded and unbounded data streams. The framework was created with the original motivation of resolving challenges faced when analyzing and processing data from the physical economy, including streams of data generated by IoT and enterprise systems. These required rapid reaction while calling for the application of advanced computation paradigms (machinelearning-powered analytics, contextual analysis, and other elements of complex event processing). Pathway is equipped with a Table API tailored for Python and Python/SQL workflows, and is powered by a distributed incremental dataflow in Rust. We describe the system and present benchmarking results which demonstrate its capabilities in both batch and streaming contexts, where it is able to surpass state-of-the-art industry frameworks in both scenarios. We also discuss streaming use cases handled by Pathway which cannot be easily resolved with state-of-the-art industry frameworks, such as streaming iterative graph algorithms (PageRank, etc.).