Training Transformers in Cosine Coefficient Space
This provides a method for reducing storage and computational costs in transformers, which is incremental as it builds on existing compression techniques without architectural changes.
The paper tackles the problem of compressing transformer models by parameterizing weight matrices in the discrete cosine transform domain, retaining only low-frequency coefficients, and achieves a 52% parameter reduction with matching perplexity (6.1 vs. 6.1) and outperforms a low-rank baseline at 4x compression (6.9 vs. 8.8 perplexity).
We parameterize the weight matrices of a transformer in the two-dimensional discrete cosine transform (DCT) domain, retaining only the lowest-frequency coefficients. At each forward pass the full weight matrix is reconstructed via the inverse DCT; gradients propagate through the reconstruction to update the spectral coefficients directly. On character-level language modeling (Shakespeare, 1M characters), a 4-layer transformer trained from scratch in this representation matches the perplexity of the standard parameterization (6.1 vs.\ 6.1) while storing 52\% of the parameters. At 4$\times$ compression (29\% of parameters), the model reaches perplexity 6.9 -- outperforming a low-rank baseline (perplexity 8.8 at 21\% of parameters) at a comparable reduction. The method requires no architectural changes, no pre-trained checkpoint, and no auxiliary loss. It reduces to replacing each \texttt{nn.Linear} with a drop-in spectral layer that stores $K$ DCT coefficients instead of $n \times m$ weights.