A Workflow Manager for Complex NLP and Content Curation Pipelines
This addresses the problem of managing complex NLP and content curation workflows for industry practitioners, but it appears incremental as it builds on existing workflow management concepts.
The authors tackled the challenge of interoperability and resource usage in NLP processing pipelines by developing a workflow manager based on principles of generality, flexibility, scalability, and efficiency, with implementation grounded in real-world industry use cases.
We present a workflow manager for the flexible creation and customisation of NLP processing pipelines. The workflow manager addresses challenges in interoperability across various different NLP tasks and hardware-based resource usage. Based on the four key principles of generality, flexibility, scalability and efficiency, we present the first version of the workflow manager by providing details on its custom definition language, explaining the communication components and the general system architecture and setup. We currently implement the system, which is grounded and motivated by real-world industry use cases in several innovation and transfer projects.