Simplicity is Key: An Unsupervised Pretraining Approach for Sparse Radio Channels
This addresses the challenge of labeling costs and flexibility in radio-based tasks, though it appears incremental as it builds on existing unsupervised and compressed sensing ideas.
The paper tackles the problem of learning representations for radio channels by introducing SpaRTan, an unsupervised pretraining approach based on compressed sensing, which reduces errors by up to 85% compared to state-of-the-art methods when fine-tuned on radio fingerprinting.
We introduce the Sparse pretrained Radio Transformer (SpaRTran), an unsupervised representation learning approach based on the concept of compressed sensing for radio channels. Our approach learns embeddings that focus on the physical properties of radio propagation, to create the optimal basis for fine-tuning on radio-based downstream tasks. SpaRTran uses a sparse gated autoencoder that induces a simplicity bias to the learned representations, resembling the sparse nature of radio propagation. For signal reconstruction, it learns a dictionary that holds atomic features, which increases flexibility across signal waveforms and spatiotemporal signal patterns. Our experiments show that SpaRTran reduces errors by up to 85 % compared to state-of-the-art methods when fine-tuned on radio fingerprinting, a challenging downstream task. In addition, our method requires less pretraining effort and offers greater flexibility, as we train it solely on individual radio signals. SpaRTran serves as an excellent base model that can be fine-tuned for various radio-based downstream tasks, effectively reducing the cost for labeling. In addition, it is significantly more versatile than existing methods and demonstrates superior generalization.