Text2Weight: Bridging Natural Language and Neural Network Weight Spaces
This work addresses the challenge of generalization to unseen tasks in neural network weight generation, advancing the practicality of generative models for parameter synthesis.
The paper tackles the problem of automatically generating neural network weights from natural language descriptions by proposing T2W, a diffusion transformer framework that produces task-specific weights, outperforming optimization-based initialization on datasets like Cifar100, Caltech256, and TinyImageNet.
How far are we really from automatically generating neural networks? While neural network weight generation shows promise, current approaches struggle with generalization to unseen tasks and practical application exploration. To address this, we propose T2W, a diffusion transformer framework that generates task-specific weights conditioned on natural language descriptions. T2W hierarchically processes network parameters into uniform blocks, integrates text embeddings from CLIP via a prior attention mechanism, and employs adversarial training with weight-space augmentation to enhance generalization. Experiments on Cifar100, Caltech256, and TinyImageNet demonstrate T2W's ability to produce high-quality weights for unseen tasks, outperforming optimization-based initialization and enabling novel applications such as weight enhancement and text-guided model fusion. Our work bridges textual semantics with weight-space dynamics, supported by an open-source dataset of text-weight pairs, advancing the practicality of generative models in neural network parameter synthesis. Our code is available on Github.