I. Oseledets

L. Markeeva, I. Tsybulin, I. Oseledets

The goal of this paper is to develop a numerical algorithm that solves a two-dimensional elliptic partial differential equation in a polygonal domain using tensor methods and ideas from isogeometric analysis. The proposed algorithm is based on the Finite Element (FE) approximation with Quantized Tensor Train decomposition (QTT) used for matrix representation and solution approximation. In this paper we propose a special discretisation scheme that allows to construct the global stiffness matrix in the QTT-format. The algorithm has $O(\log n)$ complexity, where $n=2^d$ is the number of nodes per quadrangle side. A new operation called z-kron is introduced for QTT-format. It makes it possible to build a matrix in z-order if the matrix can be expressed in terms of Kronecker products and sums. An algorithm for building a QTT coefficient matrix for FEM in z-order "on the fly", as opposed to the transformation of a calculated matrix into QTT, is presented. This algorithm has $O(\log n)$ complexity for $n$ as above.

V. Abronin, A. Naumov, D. Mazur et al.

We introduce TQCompressor, a novel method for neural network model compression with improved tensor decompositions. We explore the challenges posed by the computational and storage demands of pre-trained language models in NLP tasks and propose a permutation-based enhancement to Kronecker decomposition. This enhancement makes it possible to reduce loss in model expressivity which is usually associated with factorization. We demonstrate this method applied to the GPT-2$_{small}$. The result of the compression is TQCompressedGPT-2 model, featuring 81 mln. parameters compared to 124 mln. in the GPT-2$_{small}$. We make TQCompressedGPT-2 publicly available. We further enhance the performance of the TQCompressedGPT-2 through a training strategy involving multi-step knowledge distillation, using only a 3.1% of the OpenWebText. TQCompressedGPT-2 surpasses DistilGPT-2 and KnGPT-2 in comparative evaluations, marking an advancement in the efficient and effective deployment of models in resource-constrained environments.

E. Muravleva, I. Oseledets, D. Koroteev

We present a method to construct reduced-order models for duct flows of Bingham media. Our method is based on proper orthogonal decomposition (POD) to find a low-dimensional approximation to the velocity and artificial neural network to approximate the coefficients of a given solution in the constructed POD basis. We use well-established augmented Lagrangian method and finite-element discretization in the "offline" stage. We show that the resulting approximation has a reasonable accuracy, but the evaluation of the approximate solution several orders of magnitude times faster.