ASYNC: A Cloud Engine with Asynchrony and History for Distributed Machine Learning
This addresses a practical bottleneck for practitioners in distributed ML by providing a framework to experiment with asynchronous methods, though it is incremental as it builds on existing optimization techniques.
The paper tackles the lack of robust support for asynchrony and history in distributed machine learning by introducing ASYNC, a cloud engine framework that enables implementation of asynchronous optimization methods, demonstrating it with variants of stochastic gradient descent and SAGA.
ASYNC is a framework that supports the implementation of asynchrony and history for optimization methods on distributed computing platforms. The popularity of asynchronous optimization methods has increased in distributed machine learning. However, their applicability and practical experimentation on distributed systems are limited because current bulk-processing cloud engines do not provide a robust support for asynchrony and history. With introducing three main modules and bookkeeping system-specific and application parameters, ASYNC provides practitioners with a framework to implement asynchronous machine learning methods. To demonstrate ease-of-implementation in ASYNC, the synchronous and asynchronous variants of two well-known optimization methods, stochastic gradient descent and SAGA, are demonstrated in ASYNC.