Re-Simulation-based Self-Supervised Learning for Pre-Training Foundation Models
This addresses the problem of developing effective SSL strategies for fields relying on stochastic simulators, such as high-energy physics, though it appears incremental as it adapts existing SSL methods to a specific domain.
The paper tackles the challenge of adapting self-supervised learning (SSL) to specific data types and tasks by proposing RS3L, a simulation-based SSL strategy that uses re-simulation for data augmentation in physical sciences, showing it enables powerful performance in downstream tasks like object discrimination and uncertainty mitigation.
Self-Supervised Learning (SSL) is at the core of training modern large machine learning models, providing a scheme for learning powerful representations that can be used in a variety of downstream tasks. However, SSL strategies must be adapted to the type of training data and downstream tasks required. We propose RS3L ("Re-simulation-based self-supervised representation learning"), a novel simulation-based SSL strategy that employs a method of re-simulation to drive data augmentation for contrastive learning in the physical sciences, particularly, in fields that rely on stochastic simulators. By intervening in the middle of the simulation process and re-running simulation components downstream of the intervention, we generate multiple realizations of an event, thus producing a set of augmentations covering all physics-driven variations available in the simulator. Using experiments from high-energy physics, we explore how this strategy may enable the development of a foundation model; we show how RS3L pre-training enables powerful performance in downstream tasks such as discrimination of a variety of objects and uncertainty mitigation. In addition to our results, we make the RS3L dataset publicly available for further studies on how to improve SSL strategies.