WORKSWORLD: A Domain for Integrated Numeric Planning and Scheduling of Distributed Pipelined Workflows
This addresses the challenge of integrated planning and scheduling for distributed workflows, such as ingest pipelines, offering a domain-specific solution that is incremental in its approach.
The authors tackled the problem of automated planning and scheduling for distributed data pipelines by introducing WORKSWORLD, a domain for numeric planners that jointly builds workflow graphs and schedules components, achieving solutions for linear-chain workflows with up to 14 components across eight sites using limited computational resources.
This work pursues automated planning and scheduling of distributed data pipelines, or workflows. We develop a general workflow and resource graph representation that includes both data processing and sharing components with corresponding network interfaces for scheduling. Leveraging these graphs, we introduce WORKSWORLD, a new domain for numeric domain-independent planners designed for permanently scheduled workflows, like ingest pipelines. Our framework permits users to define data sources, available workflow components, and desired data destinations and formats without explicitly declaring the entire workflow graph as a goal. The planner solves a joint planning and scheduling problem, producing a plan that both builds the workflow graph and schedules its components on the resource graph. We empirically show that a state-of-the-art numeric planner running on commodity hardware with one hour of CPU time and 30GB of memory can solve linear-chain workflows of up to 14 components across eight sites.