Automap: Towards Ergonomic Automated Parallelism for ML Models
This addresses the need for automated parallelism to reduce the expertise and experimentation required for efficient ML model training, though it appears incremental as it builds on existing compiler frameworks.
The paper tackles the challenge of efficiently partitioning large neural networks for parallel training by introducing an automated partitioner that integrates into existing compilers and workflows, recovering expert strategies like Megatron sharding for transformers.
The rapid rise in demand for training large neural network architectures has brought into focus the need for partitioning strategies, for example by using data, model, or pipeline parallelism. Implementing these methods is increasingly supported through program primitives, but identifying efficient partitioning strategies requires expensive experimentation and expertise. We present the prototype of an automated partitioner that seamlessly integrates into existing compilers and existing user workflows. Our partitioner enables SPMD-style parallelism that encompasses data parallelism and parameter/activation sharding. Through a combination of inductive tactics and search in a platform-independent partitioning IR, automap can recover expert partitioning strategies such as Megatron sharding for transformer layers.