DTM-Codec: Dynamic Token Masking for VFR Speech Coding with Efficient Boundary Selection
This work addresses the need for efficient speech coding with variable frame rates, demonstrating clear gains for neural speech codecs.
DTM-Codec introduces a dynamic token masking method for variable frame rate speech coding that improves reconstruction quality and intelligibility over fixed-frame-rate baselines under a strict matched-total-bitrate protocol.
Variable frame rate (VFR) coding has recently emerged in neural speech codecs, allocating fewer frames to redundant regions and more frames to rapidly changing speech. VFR must transmit side information about retained time steps, but prior gains are either not rigorously addressed or often minor once these overhead bits are included in total bitrate. We present Dynamic Token Masking (DTM)-Codec, a neural speech codec that demonstrates clear gains over fixed-frame-rate baselines under a strict matched-total-bitrate protocol. DTM keeps selected encoder tokens, fills masked positions with a learned <MASK> embedding, and transmits a binary keep-mask for position-aware decoding. We further introduce Path Length Equalization (PLE), a linear-time boundary selector for VFR coding that yields well-spread adaptive segments with negligible overhead. Across operating points, DTM-Codec broadly improves reconstruction quality and intelligibility over fixed-frame-rate baselines.