The Entropy of Floating-Point Numbers
This work addresses a theoretical gap in information theory for practitioners using floating-point arithmetic, but the results are incremental as they extend known relationships between differential and discrete entropy to a specific quantization scheme.
The paper provides an analytic approximation for the entropy of floating-point numbers with error bounds, showing that floating-point entropy is approximately scale-invariant and deriving closed-form expressions for common distributions.
Here we present an analytic approximation for the entropy of floating-point numbers, along with bounds on the error of this approximation. It is well-known that the differential entropy is tightly linked to the discrete entropy of a uniformly quantized random variable. Our approximation uncovers a different quantity that provides this link for floating-point quantization. Additionally, we prove that the entropy of a floating-point quantized random variable is approximately unchanged under scaling. Closed-form expressions for the floating-point entropy of common distributions are provided and compared to exact results.