REALM: An RGB and Event Aligned Latent Manifold for Cross-Modal Perception
For researchers in event-based vision, REALM provides a general-purpose cross-modal framework that eliminates task-specific training, enabling direct use of powerful RGB models for event data.
REALM learns a shared latent space between RGB and event data using LoRA to adapt event representations to frozen RGB foundation models, enabling zero-shot transfer of image-trained decoders to events. It achieves state-of-the-art performance in wide-baseline feature matching.
Event cameras provide several unique advantages over standard frame-based sensors, including high temporal resolution, low latency, and robustness to extreme lighting. However, existing learning-based approaches for event processing are typically confined to narrow, task-specific silos and lack the ability to generalize across modalities. We address this gap with REALM, a cross-modal framework that learns an RGB and Event Aligned Latent Manifold by projecting event representations into the pretrained latent space of RGB foundation models. Instead of task-specific training, we leverage low-rank adaptation (LoRA) to bridge the modality gap, effectively unlocking the geometric and semantic priors of frozen RGB backbones for asynchronous event streams. We demonstrate that REALM effectively maps events into the ViT-based foundation latent space. Our method allows us to perform downstream tasks like depth estimation and semantic segmentation by simply transferring linear heads trained on the RGB teacher. Most significantly, REALM enables the direct, zero-shot application of complex, frozen image-trained decoders, such as MASt3R, to raw event data. We demonstrate state-of-the-art performance in wide-baseline feature matching, significantly outperforming specialized architectures. Code and models are available upon acceptance.