SOM-VQ: Topology-Aware Tokenization for Interactive Generative Models
This addresses the need for interpretable and controllable discrete representations in interactive generative models, particularly for domains like human motion, music, and gesture, though it is incremental as it builds on existing vector quantization methods.
The paper tackled the problem of vector-quantized representations lacking semantic structure for interpretable human control in generative models, and introduced SOM-VQ, which learns discrete codebooks with explicit low-dimensional topology to enable direct geometric manipulation, achieving more learnable token sequences and intuitive human-in-the-loop control in domains like human motion generation.
Vector-quantized representations enable powerful discrete generative models but lack semantic structure in token space, limiting interpretable human control. We introduce SOM-VQ, a tokenization method that combines vector quantization with Self-Organizing Maps to learn discrete codebooks with explicit low-dimensional topology. Unlike standard VQ-VAE, SOM-VQ uses topology-aware updates that preserve neighborhood structure: nearby tokens on a learned grid correspond to semantically similar states, enabling direct geometric manipulation of the latent space. We demonstrate that SOM-VQ produces more learnable token sequences in the evaluated domains while providing an explicit navigable geometry in code space. Critically, the topological organization enables intuitive human-in-the-loop control: users can steer generation by manipulating distances in token space, achieving semantic alignment without frame-level constraints. We focus on human motion generation - a domain where kinematic structure, smooth temporal continuity, and interactive use cases (choreography, rehabilitation, HCI) make topology-aware control especially natural - demonstrating controlled divergence and convergence from reference sequences through simple grid-based sampling. SOM-VQ provides a general framework for interpretable discrete representations applicable to music, gesture, and other interactive generative domains.