Parameter-Free Encoders Remain Viable for RDB Foundation Models
This work clarifies the design space for RDB foundation models by demonstrating that parameter-free encoders remain viable, challenging the need for complex pre-trained encoders in many settings.
The paper investigates whether parameter-free encoders can match or exceed the performance of parameterized encoders for relational database foundation models, proving theoretical limitations of trainable parameters and showing empirically that simpler parameter-free encoders achieve strong performance across many benchmarks.
Given a relational database (RDB) storing heterogeneous tabular information, how can we predict missing (or future) values in some target column of interest? As the space of potential targets is vast across enterprise settings, it is preferable to avoid learning a new model from scratch each time there is a new prediction task. Frozen foundation models based on RDB-specific encoders provide a viable solution, but ideal design remains an open question. On the one hand, it has recently been argued that certain parameter-free subgraph encoders combined with single-table foundation models can achieve near SOTA performance, with no RDB-specific pre-training required. Meanwhile, other contemporary studies advocate for parameterized encoders pre-trained to exploit observable labels for learning task-specific representations. To address this ambiguity, we analyze RDB encoder properties specifically when labels are present as inputs, proving limitations on the potential efficacy of trainable encoder parameters. As empirical validation, we demonstrate that considerably simpler parameter-free encoders are still capable of strong performance across many relevant benchmarking tasks.