Aligning Inductive Bias for Data-Efficient Generalization in State Space Models
This work addresses the challenge of data efficiency for machine learning practitioners, offering a practical tool to enhance model performance with limited data, though it is incremental in nature.
The authors tackled the problem of data inefficiency in State Space Models (SSMs) by developing a method to align the model's inductive bias with task-specific spectral characteristics, resulting in significant improvements in generalization and sample efficiency, especially in low-data regimes.
The remarkable success of large-scale models is fundamentally tied to scaling laws, yet the finite nature of high-quality data presents a looming challenge. One of the next frontiers in modeling is data efficiency: the ability to learn more from less. A model's inductive bias is a critical lever for this, but foundational sequence models like State Space Models (SSMs) rely on a fixed bias. This fixed prior is sample-inefficient when a task's underlying structure does not match. In this work, we introduce a principled framework to solve this problem. We first formalize the inductive bias of linear time-invariant SSMs through an SSM-induced kernel, mathematically and empirically proving its spectrum is directly governed by the model's frequency response. Further, we propose a method of Task-Dependent Initialization (TDI): power spectrum matching, a fast and efficient method that aligns the model's inductive bias with the task's spectral characteristics before large-scale training. Our experiments on a diverse set of real-world benchmarks show that TDI significantly improves generalization and sample efficiency, particularly in low-data regimes. This work provides a theoretical and practical tool to create more data-efficient models, a crucial step towards sustainable scaling.