Hyper-Transforming Latent Diffusion Models
This work addresses the scalability and efficiency limitations in INR-based generative models for researchers and practitioners in machine learning, though it is incremental as it builds on existing latent diffusion models.
The paper tackles the problem of generating functions using Implicit Neural Representations (INRs) by integrating Transformer-based hypernetworks into latent diffusion models, resulting in improved scalability, expressiveness, and generalization over existing methods.
We introduce a novel generative framework for functions by integrating Implicit Neural Representations (INRs) and Transformer-based hypernetworks into latent variable models. Unlike prior approaches that rely on MLP-based hypernetworks with scalability limitations, our method employs a Transformer-based decoder to generate INR parameters from latent variables, addressing both representation capacity and computational efficiency. Our framework extends latent diffusion models (LDMs) to INR generation by replacing standard decoders with a Transformer-based hypernetwork, which can be trained either from scratch or via hyper-transforming: a strategy that fine-tunes only the decoder while freezing the pre-trained latent space. This enables efficient adaptation of existing generative models to INR-based representations without requiring full retraining. We validate our approach across multiple modalities, demonstrating improved scalability, expressiveness, and generalization over existing INR-based generative models. Our findings establish a unified and flexible framework for learning structured function representations.