Lossless Transformations and Excess Risk Bounds in Statistical Inference
This work addresses fundamental limits in statistical inference for researchers, offering theoretical tools to assess transformation impact, though it is incremental in extending existing risk analysis.
The paper characterizes lossless transformations in statistical inference, where the excess risk is zero, and develops a consistent test for such transformations. It also provides information-theoretic bounds on excess risk and introduces delta-lossless transformations, with applications across various domains.
We study the excess minimum risk in statistical inference, defined as the difference between the minimum expected loss in estimating a random variable from an observed feature vector and the minimum expected loss in estimating the same random variable from a transformation (statistic) of the feature vector. After characterizing lossless transformations, i.e., transformations for which the excess risk is zero for all loss functions, we construct a partitioning test statistic for the hypothesis that a given transformation is lossless and show that for i.i.d. data the test is strongly consistent. More generally, we develop information-theoretic upper bounds on the excess risk that uniformly hold over fairly general classes of loss functions. Based on these bounds, we introduce the notion of a delta-lossless transformation and give sufficient conditions for a given transformation to be universally delta-lossless. Applications to classification, nonparametric regression, portfolio strategies, information bottleneck, and deep learning, are also surveyed.