Compositional Sparsity as an Inductive Bias for Neural Architecture Design
For practitioners seeking efficient and interpretable neural architectures, this work provides a principled method to induce sparsity that reduces parameter count and hyperparameter sensitivity without sacrificing performance.
The paper investigates compositional sparsity as an inductive bias for neural architecture design, combining Information Filtering Networks with Homological Neural Networks to produce sparse neural graphs. HNNs match or outperform dense baselines across real-world datasets while using far fewer parameters and exhibiting lower variance.
Identifying the structural priors that enable Deep Neural Networks (DNNs) to overcome the curse of dimensionality is a fundamental challenge in machine learning theory. Existing literature suggests that effective high-dimensional learning is driven by compositional sparsity, where target functions decompose into constituents supported on low-dimensional variable subsets. To investigate this hypothesis, we combine Information Filtering Networks (IFNs), which extract sparse dependency structures via constrained information maximisation, with Homological Neural Networks (HNNs), which map the inferred topology into fixed-wiring sparse neural graphs. We formalise the design principles underlying this construction and present an interpretable pipeline in which abstraction emerges through hierarchical composition. HNNs are orders of magnitude sparser than standard DNNs and require only minimal hyperparameter tuning. On synthetic tasks with known sparse hierarchies, HNNs recover the underlying compositional structure and remain stable in regimes where dense alternatives degrade as dimensionality increases. Across a broad suite of real-world datasets, HNNs consistently match or outperform dense baselines while using far fewer parameters, exhibiting lower variance and showing reduced sensitivity to hyperparameters.