Sequential learning on a Tensor Network Born machine with Trainable Token Embedding
This work addresses the challenge of improving expressiveness and correlation modeling in quantum-inspired generative models for complex sequence data, representing an incremental advancement in the Born machine paradigm.
The study tackled the problem of enhancing generative models for sequence data by introducing trainable token embeddings in a Tensor Network Born machine, replacing static tensor indices with positive operator valued measurements. The result was a significant reduction in negative log likelihood on RNA data, outperforming one-hot embeddings and GPT2 in single-site estimation while achieving competitive correlation modeling.
Generative models aim to learn the probability distributions underlying data, enabling the generation of new, realistic samples. Quantum inspired generative models, such as Born machines based on the matrix product state framework, have demonstrated remarkable capabilities in unsupervised learning tasks. This study advances the Born machine paradigm by introducing trainable token embeddings through positive operator valued measurements, replacing the traditional approach of static tensor indices. Key technical innovations include encoding tokens as quantum measurement operators with trainable parameters and leveraging QR decomposition to adjust the physical dimensions of the MPS. This approach maximizes the utilization of operator space and enhances the model's expressiveness. Empirical results on RNA data demonstrate that the proposed method significantly reduces negative log likelihood compared to one hot embeddings, with higher physical dimensions further enhancing single site probabilities and multi site correlations. The model also outperforms GPT2 in single site estimation and achieves competitive correlation modeling, showcasing the potential of trainable POVM embeddings for complex data correlations in quantum inspired sequence modeling.