PixelBytes: Catching Unified Embedding for Multimodal Generation
This work addresses the challenge of integrating different data types for multimodal generation, but it appears incremental as it builds on existing sequence models and focuses on a specific dataset.
The paper tackled the problem of unified multimodal representation learning by introducing PixelBytes Embedding, which captures diverse inputs in a cohesive representation to enable emergent properties for generating text and pixelated images, with experiments on a specialized dataset showing that bidirectional models with PxBy embedding and convolutional layers can generate coherent multimodal sequences.
This report introduces PixelBytes Embedding, a novel approach for unified multimodal representation learning. Our method captures diverse inputs in a single, cohesive representation, enabling emergent properties for multimodal sequence generation, particularly for text and pixelated images. Inspired by state-of-the-art sequence models such as Image Transformers, PixelCNN, and Mamba-Bytes, PixelBytes aims to address the challenges of integrating different data types. We explore various model architectures, including Recurrent Neural Networks (RNNs), State Space Models (SSMs), and Attention-based models, focusing on bidirectional processing and our innovative PxBy embedding technique. Our experiments, conducted on a specialized PixelBytes Pok{é}mon dataset, demonstrate that bidirectional sequence models with PxBy embedding and convolutional layers can generate coherent multimodal sequences. This work contributes to the advancement of integrated AI models capable of understanding and generating multimodal data in a unified manner.