mPyPl: Python Monadic Pipeline Library for Complex Functional Data Processing
This library addresses data preparation challenges for developers working with complex datasets, though it is incremental as it builds on existing functional programming concepts.
The authors introduced mPyPl, a Python library that simplifies complex data processing by using a functional approach with lazy data streams, and demonstrated its application in deep learning tasks for event detection in video.
In this paper, we present a new Python library called mPyPl, which is intended to simplify complex data processing tasks using functional approach. This library defines operations on lazy data streams of named dictionaries represented as generators (so-called multi-field datastreams), and allows enriching those data streams with more 'fields' in the process of data preparation and feature extraction. Thus, most data preparation tasks can be expressed in the form of neat linear 'pipeline', similar in syntax to UNIX pipes, or |> functional composition operator in F#. We define basic operations on multi-field data streams, which resemble classical monadic operations, and show similarity of the proposed approach to monads in functional programming. We also show how the library was used in complex deep learning tasks of event detection in video, and discuss different evaluation strategies that allow for different compromises in terms of memory and performance.