How the Optimizer Shapes Learned Solutions in Equivariant Neural Networks
For researchers working on equivariant neural networks, this work highlights the underexplored role of optimizer choice in shaping learned solutions, offering practical guidance for improving performance.
The paper investigates how the choice of optimizer (Muon vs. Adam) affects the performance and learned representations in equivariant neural networks. On ModelNet40, Muon consistently outperforms Adam across architectures, yielding checkpoints with higher Hessian curvature but more regular loss surfaces and higher effective ranks in weights and representations.
Equivariant neural networks encode geometric symmetries by construction, yet they are often difficult to optimize and can underperform less constrained architectures. A growing body of work addresses this through architectural modifications such as constraint relaxation or approximate equivariance, while the role of the optimizer remains comparatively underexplored. We study this direction by comparing Muon and Adam across several equivariant and geometric architectures under pointcloud and molecular learning settings. On ModelNet40, where the comparison is clearest, Muon consistently improves over Adam across all architectures considered. We then analyze the trained ModelNet40 checkpoints through Hessian estimates, loss surface visualizations, and spectral properties of learned weights and intermediate representations. The checkpoints reached by Muon have larger Hessian curvature summaries but more regular loss surfaces, and their learned weights and representations have higher stable and effective ranks. These observations suggest that the interaction between optimizer design and geometric inductive bias deserves further attention from the community.