Gain-loss ratio of storing intermediate data from workflows
This is an incremental improvement for users building data processing pipelines in workflow management systems.
The paper tackles the problem of automatically suggesting which intermediate datasets to use and store in workflow management systems by mining association rules from existing workflows, and demonstrates that this technique can suggest subsequent modules with gain-loss ratios.
Sequentially, the systematic processing of a significant amount of data can be necessary for input datasets to get desired outputs. In a workflow management system(WMS), usually, users build workflows by manually selecting and interconnecting various modules concerning some particular tasks. Thus, a system of automatically suggesting the appropriate intermediate datasets for modules and a suggestion for the decision of saving intermediate states will be helpful in a pipeline building process. This work investigates a technique for both automatically suggesting the intermediate datasets to use and store through mining and analyzing association rules from the previously developed workflows. Investigation on workflows shows that the association rule mining technique can help us to suggest subsequent modules for retrieving and storing data and explain them with gain-loss ratios.