Generalization Bounds for Equivariant Networks on Markov Data
This work addresses a gap in generalization theory for machine learning practitioners dealing with symmetric and dependent data, but it is incremental as it extends existing bounds to incorporate Markov properties.
The paper tackles the problem of deriving generalization bounds for equivariant neural networks on Markov data, which has dependencies, by applying a new McDiarmid's inequality and group theory to compute covering numbers, resulting in a bound that guides the selection of low-dimensional irreducible representations to enhance generalization for fixed-width networks.
Equivariant neural networks play a pivotal role in analyzing datasets with symmetry properties, particularly in complex data structures. However, integrating equivariance with Markov properties presents notable challenges due to the inherent dependencies within such data. Previous research has primarily concentrated on establishing generalization bounds under the assumption of independently and identically distributed data, frequently neglecting the influence of Markov dependencies. In this study, we investigate the impact of Markov properties on generalization performance alongside the role of equivariance within this context. We begin by applying a new McDiarmid's inequality to derive a generalization bound for neural networks trained on Markov datasets, using Rademacher complexity as a central measure of model capacity. Subsequently, we utilize group theory to compute the covering number under equivariant constraints, enabling us to obtain an upper bound on the Rademacher complexity based on this covering number. This bound provides practical insights into selecting low-dimensional irreducible representations, enhancing generalization performance for fixed-width equivariant neural networks.