Fast Tensorization of Neural Networks via Slice-wise Feature Distillation
Provides a more efficient and accurate method for compressing large neural networks, addressing the bottleneck of costly global finetuning in tensor decomposition.
Proposed a scalable tensorization framework for neural network compression that decomposes networks into slices and tensorizes each independently via feature distillation, achieving near-lossless compression on ResNet-34 and demonstrating scalability on GPT-2 XL.
We propose a scalable tensorization framework for neural network compression based on slice-wise feature distillation. Unlike conventional tensor decomposition methods that rely on costly global finetuning, our approach decomposes the network into slices consisting of either individual layers or blocks (e.g., convolutional layers or MLPs), or small groups of consecutive layers, and tensorizes each slice independently to reproduce the intermediate representations of the original pretrained model. This modular strategy improves accuracy recovery, reduces data requirements, and enables efficient parallel optimization. Experiments on ResNet-34 show significant gains over conventional global tensorization, achieving near-lossless compression at moderate compression rates with faster optimization. Results on GPT-2 XL further demonstrate the scalability of the method and its applicability to large-scale models, particularly in distributed settings.