Embarrassingly Parallel Independent Training of Multi-Layer Perceptrons with Heterogeneous Architectures
This addresses the problem of time-consuming neural network architecture search for researchers and practitioners, though it is incremental as it builds on existing parallelization techniques.
The paper tackles the challenge of efficiently training multiple independent MLPs with varying architectures by proposing ParallelMLPs, a method that uses modified matrix multiplication to enable parallel training on CPUs and GPUs, achieving a training speedup of 1 to 4 orders of magnitude compared to sequential approaches.
The definition of a Neural Network architecture is one of the most critical and challenging tasks to perform. In this paper, we propose ParallelMLPs. ParallelMLPs is a procedure to enable the training of several independent Multilayer Perceptron Neural Networks with a different number of neurons and activation functions in parallel by exploring the principle of locality and parallelization capabilities of modern CPUs and GPUs. The core idea of this technique is to use a Modified Matrix Multiplication that replaces an ordinal matrix multiplication by two simple matrix operations that allow separate and independent paths for gradient flowing, which can be used in other scenarios. We have assessed our algorithm in simulated datasets varying the number of samples, features and batches using 10,000 different models. We achieved a training speedup from 1 to 4 orders of magnitude if compared to the sequential approach.