SysML'19 demo: customizable and reusable Collective Knowledge pipelines to automate and reproduce machine learning experiments
This addresses the tedious and ad hoc process of reproducing ML results for researchers and practitioners, though it is incremental as it builds on existing automation tools.
The authors tackled the problem of reproducing and comparing machine learning experiments by introducing customizable Collective Knowledge pipelines, which automate benchmarking and co-design of software/hardware stacks, as demonstrated with real-world workflows from SysML'19, companies, and MLPerf.
Reproducing, comparing and reusing results from machine learning and systems papers is a very tedious, ad hoc and time-consuming process. I will demonstrate how to automate this process using open-source, portable, customizable and CLI-based Collective Knowledge workflows and pipelines developed by the community. I will help participants run several real-world non-virtualized CK workflows from the SysML'19 conference, companies (General Motors, Arm) and MLPerf benchmark to automate benchmarking and co-design of efficient software/hardware stacks for machine learning workloads. I hope that our approach will help authors reduce their effort when sharing reusable and extensible research artifacts while enabling artifact evaluators to automatically validate experimental results from published papers in a standard and portable way.