Entity Embeddings : Perspectives Towards an Omni-Modality Era for Large Language Models
This is an incremental vision for improving multimodal AI systems, addressing limitations in current models for researchers and developers.
The paper tackles the problem of integrating multiple modalities into large language models by proposing a framework where conceptual entities in text are treated as implicit modalities, aiming to overcome cognitive and computational limitations, but it does not provide concrete results or numbers.
Large Language Models (LLMs) are evolving to integrate multiple modalities, such as text, image, and audio into a unified linguistic space. We envision a future direction based on this framework where conceptual entities defined in sequences of text can also be imagined as modalities. Such a formulation has the potential to overcome the cognitive and computational limitations of current models. Several illustrative examples of such potential implicit modalities are given. Along with vast promises of the hypothesized structure, expected challenges are discussed as well.