TensorBank: Tensor Lakehouse for Foundation Model Training
This addresses a critical bottleneck for researchers and engineers training foundation models across domains like geospatial-temporal data, computer vision, and computational neuroscience, though it is an incremental improvement leveraging existing open standards and technology.
The paper tackles the problem of efficiently storing and streaming high-dimensional tensor data for foundation model training by introducing TensorBank, a petabyte-scale tensor lakehouse that streams tensors from cloud object storage to GPU memory at wire speed using hierarchical statistical indices for query acceleration.
Storing and streaming high dimensional data for foundation model training became a critical requirement with the rise of foundation models beyond natural language. In this paper we introduce TensorBank, a petabyte scale tensor lakehouse capable of streaming tensors from Cloud Object Store (COS) to GPU memory at wire speed based on complex relational queries. We use Hierarchical Statistical Indices (HSI) for query acceleration. Our architecture allows to directly address tensors on block level using HTTP range reads. Once in GPU memory, data can be transformed using PyTorch transforms. We provide a generic PyTorch dataset type with a corresponding dataset factory translating relational queries and requested transformations as an instance. By making use of the HSI, irrelevant blocks can be skipped without reading them as those indices contain statistics on their content at different hierarchical resolution levels. This is an opinionated architecture powered by open standards and making heavy use of open-source technology. Although, hardened for production use using geospatial-temporal data, this architecture generalizes to other use case like computer vision, computational neuroscience, biological sequence analysis and more.