From Physics to Representation: Audio Learning with Synthetic Pre-training via Procedural Generation
For audio representation learning, this work reduces reliance on large real-world datasets, offering an efficient and interpretable alternative when such data is unavailable.
AudioPG uses procedural synthesis to generate synthetic audio for pre-training a Transformer-based masked autoencoder, eliminating the need for real recordings. It achieves strong results on benchmarks (e.g., 90.60% on ESC-50) while pre-training in under 20 minutes on a single GPU.
Self-supervised learning advances audio representation for multimedia analysis. However, prevailing data-centric approaches rely on massive real-world corpora, increasing training costs, curation burdens, and privacy barriers. To address this, we present AudioPG, a procedural synthesis framework eliminating real audio recordings during pre-training. AudioPG trains a Transformer-based masked autoencoder on waveforms generated on-the-fly from basic acoustic primitives and composition rules. The encoder transfers effectively to real audio benchmarks, achieving 90.60% accuracy on ESC-50, 0.546 mAP on FSD50K, 88.17% on UrbanSound8K, and 97.03% on Speech Commands V2. Notably, pre-training completes in under 20 minutes on a single GPU. Latent space analysis reveals physical factors, including fundamental frequency and relative intensity, emerge in orthogonal subspaces, making representations linearly decodable. These results establish procedural synthesis as an efficient, interpretable pre-training signal when large-scale corpora are unavailable. Our code is available at: https://github.com/Freyliu0516/audioPG.