Comprehensive Plugin-Based Monitoring of Nexflow Workflow Executions
This provides bioinformatics and earth observation researchers with improved workflow monitoring capabilities for performance analysis, debugging, and data provenance.
The authors tackled the challenge of monitoring Nextflow workflow executions by developing a plugin-based monitoring system that provides more detailed and flexible monitoring than existing tools while maintaining compatibility with wf-commons, eliminating the need for custom Nextflow forks through Nextflow's plug-in mechanism.
Nextflow is a workflow management system commonly used in fields like bioinformatics and earth observation. It coordinates distributed data processing of various tools as an acyclic sequence of tasks while using, containerization (e.g., Docker), orchestration (e.g., Kubernetes), or batch processing (e.g., SLURM). Monitoring such workflow executions can be challenging but aids performance analysis, debugging, and data provenance. Besides Nexflow's basic built-in monitoring, the wf-commons tool for creating wf-instances is widely regarded as the standard in the Nextflow community. The monitoring plugin we develpoed provides a more detailed and flexible alternative compatible with wf-instances while removing the need for a custom Nextflow fork by using Nextflow's plug-in mechanism (version 21.10), optional direct .jar file changes of static artifacts without recompilation and allows online monitoring during execution.