Discrete JEPA: Learning Discrete Token Representations without Reconstruction
This work addresses a domain-specific problem in AI for symbolic world modeling and planning, though it is an initial model.
The paper tackled the problem of image tokenization methods lacking symbolic abstraction and logical reasoning for systematic inference, and proposed Discrete-JEPA, which outperforms baselines on visual symbolic prediction tasks with evidence of emergent systematic patterns in the token space.
The cornerstone of cognitive intelligence lies in extracting hidden patterns from observations and leveraging these principles to systematically predict future outcomes. However, current image tokenization methods demonstrate significant limitations in tasks requiring symbolic abstraction and logical reasoning capabilities essential for systematic inference. To address this challenge, we propose Discrete-JEPA, extending the latent predictive coding framework with semantic tokenization and novel complementary objectives to create robust tokenization for symbolic reasoning tasks. Discrete-JEPA dramatically outperforms baselines on visual symbolic prediction tasks, while striking visual evidence reveals the spontaneous emergence of deliberate systematic patterns within the learned semantic token space. Though an initial model, our approach promises a significant impact for advancing Symbolic world modeling and planning capabilities in artificial intelligence systems.