Drawback of Enforcing Equivariance and its Compensation via the Lens of Expressive Power
This addresses a theoretical gap in understanding the trade-offs of equivariant neural networks for researchers in symmetry-based machine learning, though it is incremental as it focuses on specific network types.
The paper investigates how enforcing equivariance constraints in 2-layer ReLU networks can strictly limit expressive power, but shows this drawback can be compensated by increasing model size, leading to lower hypothesis complexity and potentially superior generalizability.
Equivariant neural networks encode symmetry as an inductive bias and have achieved strong empirical performance in wide domains. However, their expressive power remains not well understood. Focusing on 2-layer ReLU networks, this paper investigates the impact of equivariance constraints on the expressivity of equivariant and layer-wise equivariant networks. By examining the boundary hyperplanes and the channel vectors of ReLU networks, we construct an example showing that equivariance constraints could strictly limit expressive power. However, we demonstrate that this drawback can be compensated via enlarging the model size. Furthermore, we show that despite a larger model size, the resulting architecture could still correspond to a hypothesis space with lower complexity, implying superior generalizability for equivariant networks.