Pathway: a fast and flexible unified stream data processing framework
  for analytical and Machine Learning applications
Michal Bartoszkiewicz Jan Chorowski Adrian Kosowski Jakub Kowalski Sergey Kulik Mateusz Lewandowski Krzysztof Nowicki Kamil Piechowiak Olivier Ruas Zuzanna Stamirowska Przemyslaw Uznanski
We present Pathway, a new unified data processing framework that can run
workloads on both bounded and unbounded data streams. The framework was created
with the original motivation of resolving challenges faced when analyzing and
processing data from the physical economy, including streams of data generated
by IoT and enterprise systems. These required rapid reaction while calling for
the application of advanced computation paradigms (machinelearning-powered
analytics, contextual analysis, and other elements of complex event
processing). Pathway is equipped with a Table API tailored for Python and
Python/SQL workflows, and is powered by a distributed incremental dataflow in
Rust. We describe the system and present benchmarking results which demonstrate
its capabilities in both batch and streaming contexts, where it is able to
surpass state-of-the-art industry frameworks in both scenarios. We also discuss
streaming use cases handled by Pathway which cannot be easily resolved with
state-of-the-art industry frameworks, such as streaming iterative graph
algorithms (PageRank, etc.).