Rethinking the shape convention of an MLP
This work addresses the problem of inefficient skip connection placement in MLPs for researchers and practitioners, offering a novel architectural design that could extend to Transformers and other residual networks, though it is incremental as it modifies an existing convention.
The authors challenged the conventional narrow-wide-narrow design of MLPs by proposing a wide-narrow-wide (Hourglass) architecture with skip connections at expanded dimensions, and found that it consistently achieves superior performance-parameter Pareto frontiers on generative image tasks, with optimal scaling favoring deeper networks and wider skip connections as parameters increase.
Multi-layer perceptrons (MLPs) conventionally follow a narrow-wide-narrow design where skip connections operate at the input/output dimensions while processing occurs in expanded hidden spaces. We challenge this convention by proposing wide-narrow-wide (Hourglass) MLP blocks where skip connections operate at expanded dimensions while residual computation flows through narrow bottlenecks. This inversion leverages higher-dimensional spaces for incremental refinement while maintaining computational efficiency through parameter-matched designs. Implementing Hourglass MLPs requires an initial projection to lift input signals to expanded dimensions. We propose that this projection can remain fixed at random initialization throughout training, enabling efficient training and inference implementations. We evaluate both architectures on generative tasks over popular image datasets, characterizing performance-parameter Pareto frontiers through systematic architectural search. Results show that Hourglass architectures consistently achieve superior Pareto frontiers compared to conventional designs. As parameter budgets increase, optimal Hourglass configurations favor deeper networks with wider skip connections and narrower bottlenecks-a scaling pattern distinct from conventional MLPs. Our findings suggest reconsidering skip connection placement in modern architectures, with potential applications extending to Transformers and other residual networks.