Mimetic Initialization of MLPs
This work addresses initialization for MLPs in vision tasks, but it is incremental as it extends an existing method to a new layer type with a small positive effect.
The paper tackled the problem of initializing multilayer perceptrons (MLPs) by applying mimetic initialization, which uses pretrained models to inspire simple techniques, resulting in a method that speeds up training on small-scale vision tasks like CIFAR-10 and ImageNet-1k.
Mimetic initialization uses pretrained models as case studies of good initialization, using observations of structures in trained weights to inspire new, simple initialization techniques. So far, it has been applied only to spatial mixing layers, such convolutional, self-attention, and state space layers. In this work, we present the first attempt to apply the method to channel mixing layers, namely multilayer perceptrons (MLPs). Our extremely simple technique for MLPs -- to give the first layer a nonzero mean -- speeds up training on small-scale vision tasks like CIFAR-10 and ImageNet-1k. Though its effect is much smaller than spatial mixing initializations, it can be used in conjunction with them for an additional positive effect.