CodeReef: an open platform for portable MLOps, reusable automation actions and reproducible benchmarking
This addresses the problem of inefficient and non-reproducible MLOps for researchers and practitioners, offering a solution for portable automation, though it is incremental by building on existing benchmarking and packaging concepts.
The authors tackled the challenge of automating and reproducing ML model deployment across diverse systems by introducing CodeReef, an open platform that packages models as portable archives with dependencies and workflows, demonstrating its use in building and benchmarking object detection models on platforms from Raspberry Pi to data centers with results from MLPerf inference benchmarks.
We present CodeReef - an open platform to share all the components necessary to enable cross-platform MLOps (MLSysOps), i.e. automating the deployment of ML models across diverse systems in the most efficient way. We also introduce the CodeReef solution - a way to package and share models as non-virtualized, portable, customizable and reproducible archive files. Such ML packages include JSON meta description of models with all dependencies, Python APIs, CLI actions and portable workflows necessary to automatically build, benchmark, test and customize models across diverse platforms, AI frameworks, libraries, compilers and datasets. We demonstrate several CodeReef solutions to automatically build, run and measure object detection based on SSD-Mobilenets, TensorFlow and COCO dataset from the latest MLPerf inference benchmark across a wide range of platforms from Raspberry Pi, Android phones and IoT devices to data centers. Our long-term goal is to help researchers share their new techniques as production-ready packages along with research papers to participate in collaborative and reproducible benchmarking, compare the different ML/software/hardware stacks and select the most efficient ones on a Pareto frontier using online CodeReef dashboards.