On Transferring Transferability: Towards a Theory for Size Generalization
This work addresses the challenge of dimension-independent learning for tasks with varying input sizes, such as graphs and point clouds, offering a theoretical foundation and practical guidelines for designing transferable models.
The paper tackles the problem of transferring model performance across input sizes by introducing a general framework for transferability across dimensions, showing that it corresponds to continuity in a limit space, and provides design principles and numerical experiments to support the findings.
Many modern learning tasks require models that can take inputs of varying sizes. Consequently, dimension-independent architectures have been proposed for domains where the inputs are graphs, sets, and point clouds. Recent work on graph neural networks has explored whether a model trained on low-dimensional data can transfer its performance to higher-dimensional inputs. We extend this body of work by introducing a general framework for transferability across dimensions. We show that transferability corresponds precisely to continuity in a limit space formed by identifying small problem instances with equivalent large ones. This identification is driven by the data and the learning task. We instantiate our framework on existing architectures, and implement the necessary changes to ensure their transferability. Finally, we provide design principles for designing new transferable models. Numerical experiments support our findings.