Wavelet as Tokenizer: Preliminary Results on a Shared Wavelet Token Schema for Natural Signals
This work explores a unified tokenization approach for multiple natural signal modalities, but results are preliminary and do not yet establish a universal discrete vocabulary.
The paper proposes a shared wavelet token schema for audio, images, and video, achieving PSNR of 39.92 dB (audio), 29.37 dB (image), and 23.93 dB (video) with a dense model, and shows that energy-based token selection provides significant gains over uniform selection (e.g., 16.73 dB for audio).
This paper studies whether audio, images, and video can share a common wavelet token schema rather than relying on separate modality-specific latent grids. It introduces a preliminary continuous-token model built around a one-level Haar DWT/IDWT frontend, a shared coefficient-token layout, optional structural metadata, lightweight modality value adapters, and a shared token-wise encoder-decoder trunk. On Speech Commands, EuroSAT RGB, and DAVIS 2017 data, a dense shared model reaches 39.92 dB audio, 29.37 dB image, and 23.93 dB video PSNR. A matched-rate sweep under continuous latent scalar budgets indicates that the visual gains are not explained solely by latent capacity, while also showing that additive metadata embeddings are not a universal source of improvement. Finally, fixed-rate energy selection provides a strong non-parametric baseline: energy_global improves average PSNR over uniform selection by 16.73 dB for audio, 16.90 dB for images, and 15.86 dB for video under compressed keep ratios. Masked sparse training reaches 34.45 dB video PSNR with 50% of dense tokens. The results support a unified wavelet token schema and sparse token interface, while stopping short of establishing a universal discrete vocabulary.