multivariateGPT: a decoder-only transformer for multivariate categorical and numeric data
This work addresses the challenge of handling mixed categorical and numeric data in sequences for applications such as healthcare and time series analysis, representing an incremental advancement by adapting existing transformer architectures.
The authors tackled the problem of modeling sequences containing both categorical and numeric data, which often occur in real-world processes with irregular sampling, by introducing multivariateGPT, a decoder-only transformer that jointly predicts next token class and value. They demonstrated its ability to learn patterns in physical systems and model complex time series like electrocardiograms and health records, extending transformer utility to new data classes.
Real-world processes often generate data that are a mix of categorical and numeric values that are recorded at irregular and informative intervals. Discrete token-based approaches are limited in numeric representation capacity while methods like neural ordinary differential equations are not well suited for categorical data or informative sampling and require augmentation to handle certain classes of trajectories. Here, we present multivariateGPT, a single architecture for modeling sequences of mixed categorical (including tokenized text) and numeric data. This is accomplished with an autoregressive sequence decomposition, embedding scheme, and loss function that extend the next token prediction task to likelihood estimation of the joint distribution of next token class and value. We demonstrate how this approach can efficiently learn to generalize patterns in simple physical systems and model complex time series including electrocardiograms and multivariate electronic health record data. This work extends the utility of transformer based models to additional classes of data.