Alexey Frolov

Albert Sayapin, Gleb Balitskiy, Daniel Bershatsky et al.

In this study, we propose a novel SeqMF model to solve the problem of predicting the next app launch during mobile device usage. Although this problem can be represented as a classical collaborative filtering problem, it requires proper modification since the data are sequential, the user feedback is distributed among devices and the transmission of users' data to aggregate common patterns must be protected against leakage. According to such requirements, we modify the structure of the classical matrix factorization model and update the training procedure to sequential learning. Since the data about user experience are distributed among devices, the federated learning setup is used to train the proposed sequential matrix factorization model. One more ingredient of the proposed approach is a new privacy mechanism that guarantees the protection of the sent data from the users to the remote server. To demonstrate the efficiency of the proposed model we use publicly available mobile user behavior data. We compare our model with sequential rules and models based on the frequency of app launches. The comparison is conducted in static and dynamic environments. The static environment evaluates how our model processes sequential data compared to competitors. Therefore, the standard train-validation-test evaluation procedure is used. The dynamic environment emulates the real-world scenario, where users generate new data by running apps on devices, and evaluates our model in this case. Our experiments show that the proposed model provides comparable quality with other methods in the static environment. However, more importantly, our method achieves a better privacy-utility trade-off than competitors in the dynamic environment, which provides more accurate simulations of real-world usage.

Gennady Shutkov, Dmitry Artemasov, Alexey Frolov et al.

We consider the design of low-density parity-check (LDPC) codes for a given iterative decoder. Despite tools such as direct simulation, density evolution (DE), and EXIT-chart analysis, selecting a parity-check matrix remains a difficult combinatorial optimization problem. Existing approaches often rely on population-based search, random mutations, genetic algorithms, or related heuristics, which require careful parameter tuning and may be computationally expensive. Recent gradient descent (GD)-based methods optimize relaxed parity-check matrices by differentiating through decoder simulations. However, such decoder-in-the-loop strategies rely on noisy Monte Carlo estimates, require line search over soft matrix representations, and remain costly for long LDPC codes. Moreover, although optimization is performed in a relaxed domain, the loss is typically evaluated only at integer-valued parity-check matrices. In this work, we focus on the design of long protograph-based LDPC codes and propose a deterministic GD-based framework that operates directly on a relaxed protograph representation. Each protograph entry is interpreted as the probability that the corresponding element is equal to one. The loss function is based on DE bit error rate (BER) performance and can be evaluated directly for relaxed protographs. To justify this relaxation, we associate the relaxed representation with an ensemble of binary protographs and show that the proposed relaxed DE gives the ensemble-averaged DE performance. The resulting optimization procedure is fully autonomous and uses standard GD methods. Owing to deterministic DE evaluation and informative gradients, the proposed approach provides fast and reliable convergence. Numerical experiments for the min-sum decoder show that the optimized protographs outperform 5G LDPC codes with the same protograph dimensions.

Oleg Nesterenkov, Kirill Andreev, Alexey Frolov et al.

This paper studies low-complexity soft-output decoding of turbo product codes with extended Bose--Chaudhuri--Hocquenghem component codes. Recent soft-output from covered space (SOCS) decoding substantially improves the quality of extrinsic information compared with the conventional Chase--Pyndiah decoder, but its probabilistic-domain implementation is less attractive for hardware-oriented realizations. We therefore propose a log-domain approximation of SOCS based on max-log approach. The proposed soft-input soft-output rule replaces probability-domain operations with a piecewise-linear function of reliability gaps between competing Chase-II decoding list and out of the list hypotheses, which preserves compatibility with the standard iterative TPC decoding loop. Numerical results for a TPC built from (256,239) eBCH component codes show that the proposed decoder clearly outperforms the baseline Chase--Pyndiah decoder with the same list size and approaches the performance of SOCS decoder.

Ivan Butakov, Alexander Semenenko, Alexey Frolov et al.

We introduce a novel Mutual Information (MI) estimator that fundamentally reframes the discriminative approach. Instead of training a classifier to discriminate between joint and marginal distributions, we learn a normalizing flow that transforms one into the other. This technique produces a computationally efficient and precise MI estimate that scales well to high dimensions and across a wide range of ground-truth MI values.

Ivan Butakov, Alexander Tolmachev, Sofia Malanchuk et al.

We propose a novel approach to the problem of mutual information (MI) estimation via introducing a family of estimators based on normalizing flows. The estimator maps original data to the target distribution, for which MI is easier to estimate. We additionally explore the target distributions with known closed-form expressions for MI. Theoretical guarantees are provided to demonstrate that our approach yields MI estimates for the original data. Experiments with high-dimensional data are conducted to highlight the practical advantages of the proposed method.

Alexander Semenenko, Ivan Butakov, Alexey Frolov et al.

Sliced Mutual Information (SMI) is widely used as a scalable alternative to mutual information for measuring non-linear statistical dependence. Despite its advantages, such as faster convergence, robustness to high dimensionality, and nullification only under statistical independence, we demonstrate that SMI is highly susceptible to data manipulation and exhibits counterintuitive behavior. Through extensive benchmarking and theoretical analysis, we show that SMI saturates easily, fails to detect increases in statistical dependence (even under linear transformations designed to enhance the extraction of information), prioritizes redundancy over informative content, and in some cases, performs worse than simpler dependence measures like the correlation coefficient.

Ivan Butakov, Alexander Tolmachev, Sofia Malanchuk et al.

The Information Bottleneck (IB) principle offers an information-theoretic framework for analyzing the training process of deep neural networks (DNNs). Its essence lies in tracking the dynamics of two mutual information (MI) values: between the hidden layer output and the DNN input/target. According to the hypothesis put forth by Shwartz-Ziv & Tishby (2017), the training process consists of two distinct phases: fitting and compression. The latter phase is believed to account for the good generalization performance exhibited by DNNs. Due to the challenging nature of estimating MI between high-dimensional random vectors, this hypothesis was only partially verified for NNs of tiny sizes or specific types, such as quantized NNs. In this paper, we introduce a framework for conducting IB analysis of general NNs. Our approach leverages the stochastic NN method proposed by Goldfeld et al. (2019) and incorporates a compression step to overcome the obstacles associated with high dimensionality. In other words, we estimate the MI between the compressed representations of high-dimensional random vectors. The proposed method is supported by both theoretical and practical justifications. Notably, we demonstrate the accuracy of our estimator through synthetic experiments featuring predefined MI values and comparison with MINE (Belghazi et al., 2018). Finally, we perform IB analysis on a close-to-real-scale convolutional DNN, which reveals new features of the MI dynamics.