Parameter Database : Data-centric Synchronization for Scalable Machine Learning
This addresses synchronization bottlenecks for scalable machine learning, though it appears incremental as it builds on existing synchronization methods.
The paper tackles the problem of inefficient synchronization in distributed machine learning by proposing a data-centric framework that relaxes the bulk synchronization parallel paradigm, resulting in substantial improvements over BSP while ensuring sequential correctness.
We propose a new data-centric synchronization framework for carrying out of machine learning (ML) tasks in a distributed environment. Our framework exploits the iterative nature of ML algorithms and relaxes the application agnostic bulk synchronization parallel (BSP) paradigm that has previously been used for distributed machine learning. Data-centric synchronization complements function-centric synchronization based on using stale updates to increase the throughput of distributed ML computations. Experiments to validate our framework suggest that we can attain substantial improvement over BSP while guaranteeing sequential correctness of ML tasks.