Novel Aspects of IEEE SA P3109 Arithmetic Formats for Machine Learning
This work provides a standardized arithmetic format for machine learning practitioners and hardware vendors, enabling consistent and efficient low-precision computation.
The IEEE P3109 draft standard defines a parameterized family of binary floating-point formats for machine learning, enabling efficient representation and exception-free operations with extensive rounding modes including stochastic rounding. The standard includes a novel scale-invariant approximation measure (kappa-approximation) and mechanically verified specifications.
The IEEE P3109 draft standard defines a parameterized family of binary floating-point formats and associated operations, with a focus on facilitating machine learning. These formats allow efficient and consistent representation of values in a small number of bits. The defined formats are parameterized over width and precision in bits, signedness, and the presence of infinities. Operations are defined by decoding floating-point values to the set of closed extended reals: the reals augmented with positive and negative infinity and NaN (Not a Number). Explicit treatment of NaN and infinite operands ensures that only real arithmetic is invoked in operation definitions. Extensive rounding and saturation modes are defined; stochastic rounding is included. Operations are exception-free, accelerating throughput, with exceptional situations communicated through return values, e.g., NaN. Operations on blocks of values sharing a common scale factor are defined in terms of the underlying operations in a uniform manner. System vendors may describe approximate implementations via a novel scale-invariant measure, akin to units in the last place, called kappa-approximation. Standard function definitions and various other properties are mechanically verified and generated using formal specifications.