Is the reconstruction loss culprit? An attempt to outperform JEPA
This work addresses representation learning robustness for researchers, but it is incremental as it focuses on a toy testbed with limited real-world applicability.
The authors investigated whether reconstruction loss is the main issue in autoencoders compared to JEPA-style predictive representation learning, using a controlled linear dynamical system testbed. They found that autoencoder failures were due to objective asymmetries and bottleneck effects, and introduced a gated predictive autoencoder that matched or outperformed JEPA in stability and performance across noise levels.
We evaluate JEPA-style predictive representation learning versus reconstruction-based autoencoders on a controlled "TV-series" linear dynamical system with known latent state and a single noise parameter. While an initial comparison suggests JEPA is markedly more robust to noise, further diagnostics show that autoencoder failures are strongly influenced by asymmetries in objectives and by bottleneck/component-selection effects (confirmed by PCA baselines). Motivated by these findings, we introduce gated predictive autoencoders that learn to select predictable components, mimicking the beneficial feature-selection behavior observed in over-parameterized PCA. On this toy testbed, the proposed gated model is stable across noise levels and matches or outperforms JEPA.