Sampling Foundational Transformer: A Theoretical Perspective
This addresses the problem of foundational modeling requiring operability on diverse data sources, though it appears incremental as it builds on existing transformer frameworks with efficiency improvements.
The paper tackles the challenge of applying transformers across multiple data modalities like point clouds and graphs by proposing the Sampling Foundational Transformer (SFT), which achieves competitive results on benchmarks and faster inference times compared to specialized models.
The versatility of self-attention mechanism earned transformers great success in almost all data modalities, with limitations on the quadratic complexity and difficulty of training. To apply transformers across different data modalities, practitioners have to make specific clever data-modality-dependent constructions. In this paper, we propose Sampling Foundational Transformer (SFT) that can work on multiple data modalities (e.g., point cloud, graph, and sequence) and constraints (e.g., rotational-invariant). The existence of such model is important as contemporary foundational modeling requires operability on multiple data sources. For efficiency on large number of tokens, our model relies on our context aware sampling-without-replacement mechanism for both linear asymptotic computational complexity and real inference time gain. For efficiency, we rely on our newly discovered pseudoconvex formulation of transformer layer to increase model's convergence rate. As a model working on multiple data modalities, SFT has achieved competitive results on many benchmarks, while being faster in inference, compared to other very specialized models.