TabPATE: Differentially Private Tabular In-Context Learning Without Public Data
This work addresses the privacy vulnerability of tabular foundation models for practitioners who lack public data, offering a practical solution with strong empirical guarantees.
TabPATE introduces a differentially private defense for tabular in-context learning that eliminates the need for public in-distribution data, achieving near-random membership inference success while maintaining competitive utility across benchmarks.
Tabular foundation models enable accurate in-context learning (ICL) from small labeled datasets, but the private records placed in context can leak through model predictions. We first show that even basic membership inference attacks succeed against tabular ICL, motivating formal privacy protection. We then introduce TabPATE, a differentially private PATE-style defense for tabular ICL that does not require public in-distribution data. TabPATE partitions the private context across teacher models, privately aggregates their labels on synthetic tabular queries, and releases the resulting labeled queries as a student context. Because tabular features are bounded and relatively low-dimensional, useful queries can be generated from feature ranges alone or from lightly privatized marginals. Across tabular benchmarks, TabPATE preserves competitive utility while reducing membership inference to near-random success, providing a practical path to private tabular ICL without public data.