Toward Temporal Attribution Analytics in Dataflows
For developers and operators of large-scale stream processing systems, this work introduces a new provenance paradigm to monitor dependencies over time with reduced computational and storage costs, though it is currently a vision without empirical validation.
The paper defines temporal attribution, a lightweight form of provenance for dataflows, and proposes a state-based indexing approach to enable time-focused analysis without fine-grained metadata, aiming to address scalability challenges in streaming systems like Apache Flink.
Data provenance (the process of determining the origin and derivation of data outputs) has applications across multiple domains including explaining database query results and auditing scientific workflows. Despite decades of research, provenance tracing remains challenging due to its high computational cost and storage requirements. In streaming systems such as Apache Flink, fine-grained provenance graphs can grow super-linearly with data volume, posing significant scalability challenges. We define temporal attribution, a new lightweight form of provenance, appropriate for certain tasks, such as monitoring dependencies between system components over time quantitatively. Temporal attribution enables time-focused analysis that does not require fine-grained, tuple-level dependency meta-data. Inspired by volume-based provenance tracking in Temporal Interaction Networks (TINs), we demonstrate TINs' applicability in succinctly modeling quantified data exchanges between dataflow operators in stream data processing systems and in processing workflows, in general, over time. We classify data into discrete and liquid types, define five temporal provenance query types, and propose a state-based indexing approach. Our vision outlines research directions toward making this new form of temporal attribution a practical tool for large-scale dataflow analytics.