Masked Particle Modeling on Sets: Towards Self-Supervised High Energy Physics Foundation Models
This work addresses the need for foundation models in high energy physics, offering a novel self-supervised approach for set-based data, though it is incremental in adapting masked modeling techniques to this specific domain.
The authors tackled the problem of learning generic representations for unordered sets in high energy physics by proposing masked particle modeling, a self-supervised method that recovers masked particle identities using discretized tokens, and demonstrated its efficacy in jet classification tasks with efficient fine-tuning to new classes and domains.
We propose masked particle modeling (MPM) as a self-supervised method for learning generic, transferable, and reusable representations on unordered sets of inputs for use in high energy physics (HEP) scientific data. This work provides a novel scheme to perform masked modeling based pre-training to learn permutation invariant functions on sets. More generally, this work provides a step towards building large foundation models for HEP that can be generically pre-trained with self-supervised learning and later fine-tuned for a variety of down-stream tasks. In MPM, particles in a set are masked and the training objective is to recover their identity, as defined by a discretized token representation of a pre-trained vector quantized variational autoencoder. We study the efficacy of the method in samples of high energy jets at collider physics experiments, including studies on the impact of discretization, permutation invariance, and ordering. We also study the fine-tuning capability of the model, showing that it can be adapted to tasks such as supervised and weakly supervised jet classification, and that the model can transfer efficiently with small fine-tuning data sets to new classes and new data domains.