Astronomical Pipeline Provenance: A Use Case Evaluation
This addresses the need for reliable data handling in astronomy, particularly for large-scale projects, but is incremental as it focuses on use case analysis rather than a novel solution.
The paper tackles the challenge of managing massive data volumes from next-generation astronomical observatories like the SKA, which produce up to 10 TB/s and 600 PB/year, by proposing a foundation for an automated provenance generation tool to ensure trust and reproducibility in data processing pipelines.
In this decade astronomy is undergoing a paradigm shift to handle data from next generation observatories such as the Square Kilometre Array (SKA) or the Vera C. Rubin Observatory (LSST). Producing real time data streams of up to 10 TB/s and data products of the order of 600 Pbytes/year, the SKA will be the biggest civil data producing machine of the world that demands novel solutions on how these data volumes can be stored and analysed. Through the use of complex, automated pipelines the provenance of this real time data processing is key to establish confidence within the system, its final data products, and ultimately its scientific results. The intention of this paper is to lay the foundation for making an automated provenance generation tool for astronomical/data-processing pipelines. We therefore present a use case analysis, specific to the astronomical needs which addresses the issues of trust and reproducibility as well as other ulterior use cases which are of interest to astronomers. This analysis is subsequently used as the basis to discuss the requirements, challenges, and opportunities involved in designing both the tool and the associated provenance model.