TREET: TRansfer Entropy Estimation via Transformers
This work addresses the challenge of accurately estimating transfer entropy for stationary processes, which is significant for applications in information theory and real-world domains like physiological data analysis, though it appears incremental as it builds on existing neural estimation methods with a novel transformer-based approach.
The authors tackled the problem of estimating transfer entropy, an information-theoretic measure for directional information flow, by proposing TREET, a novel attention-based method that leverages transformers and Donsker-Varadhan representation, achieving results demonstrated through comparisons on a dedicated benchmark and applications to channel capacity estimation and real-world physiological data analysis.
Transfer entropy (TE) is an information theoretic measure that reveals the directional flow of information between processes, providing valuable insights for a wide range of real-world applications. This work proposes Transfer Entropy Estimation via Transformers (TREET), a novel attention-based approach for estimating TE for stationary processes. The proposed approach employs Donsker-Varadhan representation to TE and leverages the attention mechanism for the task of neural estimation. We propose a detailed theoretical and empirical study of the TREET, comparing it to existing methods on a dedicated estimation benchmark. To increase its applicability, we design an estimated TE optimization scheme that is motivated by the functional representation lemma, and use it to estimate the capacity of communication channels with memory, which is a canonical optimization problem in information theory. We further demonstrate how an optimized TREET can be used to estimate underlying densities, providing experimental results. Finally, we apply TREET to feature analysis of patients with Apnea, demonstrating its applicability to real-world physiological data. Our work, applied with state-of-the-art deep learning methods, opens a new door for communication problems which are yet to be solved.